Gyōgi founded this temple on the orders of Emperor Shōmu and carved the statue of Yakushi as its honzon. At that time, Buddhist law forbid women getting any closer to the temple than the hill called Hanaori no Saka. Likewise, women were not allowed on Kōyasan or the mountain of Temple 12. In fact, women at the time were not allowed on any of the sacred mountains.

One hundred years later, Kōbō Daishi's mother, Tamayori, came to visit him while he was training here. For seventeen days he performed an esoteric rite at the temple gate. At the end of the seventeen day ritual, the ban against women climbing the mountain was lifted and Kōbō Daishi's mother was allowed to enter the temple precincts. Once she entered the temple, she shaved off her hair and became a nun.

Because Kōbō Daishi took care of his mother after she became a nun, the temple came to be known as Onzanji, a temple of gratitude to one's parents.

Frederick Starr says that Kōbō Daishi's mother's bones are buried here and that Kōbō Daishi carved the statue of himself in the daishidō.