The castle's origins go back to the Sengoku period. A fortification was built at this location by the shugō of Shinano Province, Shimadachi Sadanaga of the Ogasawara clan during the Eisho era (1504-1520). This minor border post was originally called Fukashi Castle. In 1550 it was seized by the Takeda clan following the Siege of Fukashi.

Takeda Shingen appointed his retainer Baba Nobuharu as castellan. The castle was the Takeda field headquarters for their conquest of the Matsumoto Basin and as a redoubt in the constant conflict between the Takeda and the powerful Uesugi clan to the north. Following the defeat of the Takeda clan by Oda Nobunaga in 1582, the castle was surrendered to Oda Nagamasu. It was soon reassigned to Kiso Yoshimasa.

With the assassination of Oda Nobunaga in 1582, the castle was seized by Ogasawara Dosetsuzai with the backing of Uesugi Kagekatsu. His nephew, Ogasawara Sadayoshi, later pledged fealty to Tokugawa Ieyasu, and renamed the castle "Matsumoto Castle".

Following Toyotomi Hideyoshi's conquest of Odawara in 1590, Tokugawa Ieyasu was transferred from his ancestral domains to the Kantō region, and Ishikawa Kazumasa was placed in charge of Matsumoto. Kazumasa and his son Yasunaga built the tower and other parts of the castle, including the three towers: the tenshu and the small yagura in the northwest, both begun in 1590, and the Watari Yagura, the residence, the drum gate, the black gate, the Tsukimi Yagura, the moat, the innermost bailey, the second bailey, the third bailey, and the sub-floors in the castle, much as they are today. They were also instrumental in laying out the castle town and its infrastructure. It is believed much of the castle was completed by 1593–94.

During the Edo period, the Tokugawa shogunate established the Matsumoto Domain. The Ogasawara returned briefly as daimyō of Matsumoto from 1613 to 1617. They were followed by the Toda-Matsudaira clan from 1617 to 1633, Matsudaira clan from 1633 to 1638, Hotta clan from 1638 to 1642, Mizuno clan from 1642 to 1725 and by the Toda-Matsudaira clan again from 1725 to the Meiji restoration in 1868.