There was a minister called Watanabe Terumasa, who was so religious that he prayed at Daisenji every month. One day, however, he lost his life in a fight with another minister during a visit to the temple. The monks felt sorry for him and built a grave for him in Shimoyama. After a while, white foxes started to live around the grave. People believed they were his reincarnation, so they established a small shrine deifying him as Shimoyama Zenshin. In 1352, Shimoyama Zenshin sent a message to establish a new shrine and enshrine a statue of Buddha that was to be dug up from the ground. When digging the ground as requested, a statue of Kannon Bodhisattva was found. The monks put it in the new shrine and worshiped it (now on display at the Museum of Daisenji Treasures). It is thought that the location of the shrine remained unchanged since then. You can find small holes both on the right and left sides of the shrine building, which are said to be pathways for spirits. According to legend, Terumasa would take the form of a white fox and slip out of the hole at night, going around granting wishes. Okumiya (Mountain Sanctuary) of Ogamiyama Shrine also has the same holes for him to come in to ask for help when the favor is beyond his power.
The current shrine was rebuilt in 1805 and is designated as a National Important Cultural Property.