The Casa Rosada sits at the eastern end of the Plaza de Mayo, a large square which since the 1580 foundation of Buenos Aires has been surrounded by many of the most important political institutions of the city and of Argentina. The site, originally at the shoreline of the Río de la Plata, was first occupied by the "Fort of Juan Baltazar of Austria", a structure built on the orders of the founder of Buenos Aires, Captain Juan de Garay, in 1594. Its 1713 replacement by a masonry structure (the "Castle of San Miguel") complete with turrets made the spot the effective nerve center of colonial government. Following independence, President Bernardino Rivadavia had a Neoclassical portico built at the entrance in 1825, and the building remained unchanged until, in 1857, the fort was demolished in favor of a new customs building. Under the direction of British Argentine architect Edward Taylor, the Italianate structure was Buenos Aires' largest building from 1859 until the 1890s.[1][2]

The old fort's administrative annex, which survived the construction of Taylor's Customs House, was enlisted as the presidential offices by Bartolomé Mitre in the 1860s and his successor, Domingo Sarmiento, who beautified the drab building with patios, gardens and wrought-iron grillwork, had the exterior painted pink reportedly in order to defuse political tensions by mixing the red and white colors of the country's two opposing political parties: red was the color of the Federalists, while white was the color of the Unitarians.[3] An alternative explanation suggests that the original paint contained cow's blood to prevent damage from the effects of humidity. Sarmiento also authorized the construction of the Central Post Office next door in 1873, commissioning Swedish Argentine architect Carl Kihlberg, who designed this, one of the first of Buenos Aires' many examples of Second Empire architecture.[1]

Presiding over an unprecedented socio-economic boom, President Julio Roca commissioned architect Enrique Aberg to replace the cramped State House with one resembling the neighboring Central Post Office in 1882. Following works to integrate the two structures, Roca had architect Francesco Tamburini build the iconic Italianate archway between the two in 1884. The resulting State House, still known as the "Rose House", was completed in 1898 following its eastward enlargement, works which resulted in the destruction of the customs house.[1]

A Historical Museum was created in 1957 to display presidential memorabilia and selected belongings, such as sashes, batons, books, furniture, and three carriages. The remains of the former fort were partially excavated in 1984–85, and the uncovered structures were incorporated into the Museum of the Casa Rosada. Located behind the building, these works led to the rerouting of Paseo Colón Avenue, unifying the Casa Rosada with Parque Colón (Columbus Park) behind it. Plans were announced in 2009 for the restoration of surviving portions of Taylor's Customs House, as well.[4]

The Casa Rosada itself in 2006 underwent extensive renovation delayed by the 2001 economic crisis. The first phase was completed for the 2010 bicentennial of the May Revolution that led to independence, with a second phase begun in 2017.[1]