Legend states that the temple was built on sand from the eight sacred places of India, which had been given to Kōbō Daishi by his teacher Hui Kuo. It took six years to complete construction and was dedicated in 816 (or 813). Kōbō Daishi also is credited with carving the honzon. The chief priest was traditionally appointed by the emperor himself.

This location is also called Byōbugara. Kōbō Daishi parents are Yoshimichi Saeki and Tamayori-gozen. The name he was given at birth was Mao (Real Fish).

As is appropriate, this temple, and its surrounding town, is the largest on the island of Shikoku and contains the typical seven shrines and pagoda. Be sure to visit the tunnel corridor (Kaidan Meguri) in the basement of Tanjōin. You must walk the one hundred yards through the tunnel and past 88 Buddhas without a lamp. Evil people are supposed to get stuck in the tunnel. Also in the tunnel is a small square, said to mark the exact spot of Kōbō Daishi's birth.

There is controversy about this temple, though. At the time of Kōbō Daishi's birth, women were generally banned from the house during times of menstruation and childbirth as they were considered to be ritually unclean at these times. Some say that Kōbō Daishi was born at a house nearer the beach and there is a bangai temple at that location now, but there seems little doubt that he was raised in a house at this location.

At this temple one can see a grave where Mao buried his dog, a scroll written by his mother, and a statue carved by Kōbō Daishi when he was a young man. Near the back of the temple compound is the Miedo, once the house of Kōbō Daishi's mother and now a temple. Also in the back is a new meditation hall, made of highly polished marble, and a n new stupa.

In the vicinity of the temple is Konpira Shrine (also known as Konpira-san) where the famous Shintō kami Konpira is enshrined and attracts four million visitors each year. Especially fishermen, seamen, and pilots worship this shrine because it is considered the guardian of shipwrecks and voyages. It is located on the top of Mt. Zōzu (Elephant Head) at an elevation of 1,700 ft. There are 785 stone steps to the hondō.