The idea for an outdoor display of oriental buildings, open to the public on the site, originated with King Leopold II, who had been particularly impressed by the Panorama du Tour du Monde at the Paris Exposition of 1900.[1] The French architect Alexandre Marcel was commissioned in 1900 to build a Japanese pagoda (known as a Tō). It was inaugurated in 1905.[1] The Japanese Tower (French: Tour japonaise, Dutch: Japanse Toren) stands nearly 50 metres (160 ft) tall, across the road from the rest of the museum buildings.[2]

Work on a larger Chinese Pavilion (French: Pavillon chinois, Dutch: Chinees Paviljoen) began in 1905. The building was originally intended to be a restaurant, but never served the purpose for which it was intended. In 1909, with the death of Leopold II, the original plan for a museum was abandoned and the building was donated to the Belgian State where it served as part of the Trade Museum of the Ministry of Foreign Affairs.[1] From 1947 until 1989, the whole area was closed to visitors.[1] The museums' section on Japanese art is housed in a building near the Chinese Pavilion, originally intended to serve as a stable and garage for the complex.

The Chinese Pavilion and Japanese Tower were recognised as protected monuments in 2019.[3] After closing to the public in 2013 for structural reasons,[4][5] the complex was restored but not reopened to the public. The Flemish public broadcaster VRT reported in 2022 that the Federal Government had decided the previous year not to re-open the complex although no formal announcement had been made to this effect.[6]