The site of the temple was an estate of the Fujiwara clan in the 11th century. The first temple, the Daiju-in, and the still existing large pond were built in that century by Fujiwara Saneyoshi. In 1450, Hosokawa Katsumoto, another powerful warlord, acquired the land where the temple stood. He built his residence there, and founded a Zen temple, Ryōan-ji. During the Ōnin War between the clans, the temple was destroyed. Hosokawa Katsumoto died in 1473, and in 1488 his son, Hosokawa Masamoto, rebuilt the temple.

The temple served as a mausoleum for several emperors. Their tombs are grouped together in what are today known as the "Seven Imperial Tombs" at Ryōan-ji. The burial places of these emperors—Uda, Kazan, Ichijō, Go-Suzaku, Go-Reizei, Go-Sanjō, and Horikawa—would have been comparatively humble in the period after their deaths. These tombs reached their present state as a result of the 19th century restoration of imperial sepulchers (misasagi) which were ordered by Emperor Meiji.[1]

There is controversy over who built the garden and when. Most sources date it to the second half of the 15th century.[2] According to some sources, it was built by Hosokawa Katsumoto, the creator of the first temple of Ryōan-ji, between 1450 and 1473. Other sources say it was built by his son, Hosokawa Masamoto, in or around 1488.[3] Some say that the garden was built by the famous landscape painter and monk, Sōami (died 1525),[4] but this is disputed by other authors.[5] Some sources say the garden was built in the first half of the 16th century,[6] others reckon later, during the Edo period, between 1618 and 1680.[5] There is also controversy over whether the garden was built by monks, or by professional gardeners, called kawaramono, or a combination of the two. One stone in the garden has the name of two kawaramono carved into it, Hirokojirō and Kotarō.

The conclusive history, though, based on documentary sources, is as follows: Hosokawa Katsumoto (1430–1473), deputy to the shōgun, founded in 1450 the Ryōan-ji temple, but the complex was burnt down during the Ōnin War. His son Masamoto rebuilt the temple at the very end of the same century. It is not clear whether any garden was constructed at that time facing the main hall. First descriptions of a garden, clearly describing one in front of the main hall, date from 1680–1682. It is described as a composition of nine big stones laid out to represent Tiger Cubs Crossing the Water. As the garden has fifteen stones at present, it was clearly different from the garden that we see today. A great fire destroyed the buildings in 1779, and rubble of the burnt buildings was dumped in the garden. Garden writer and specialist Akisato Rito (died c. 1830) redid the garden completely on top of the rubble at the end of the eighteenth century and published a picture of his garden in his Celebrated Gardens and Sights of Kyoto (Miyako rinsen meisho zue) of 1799, showing the garden as it looks today. One big stone at the back was buried partly; it has two first names carved in it, probably names of untouchable stone workers, so called kawaramono.[7] There is no evidence of Zen monks having worked on the garden, apart from the raking of the sand.