The origin of the castle dates back to 1304 when the generation of knights Wilre built a small castle. It was built to protect the city of Vilvoorde and the Duchy of Brabant against the attacks of the Malinois. After a long period of wars and religious strife, the castle returned to a peaceful residence. The Lord of Corbais, Jan Cools, owned it since 1619. However, he was in deep debt and was forced to sell the dilapidated castle in 1631.
In 1635 the famous Antwerp painter Pieter Paul Rubens bought the castle. He had it completely redesigned to a castle in Flemish Renaissance style. From then on the castle had changed its name from Het Steen to Castle Elewijt and later to the Rubens castle.
Rubens made a lot of his most famous works in and around the castle. After his death in 1640, Peter's second wife, Helena Fourment, live at the castle for almost the rest of her life. In 1792 the castle was converted into a state prison. The castle became a UNESCO world heritage monument in 1948 and is now privately owned property and mostly not open to the public. However, some parts of the castle can be rented for parties and seminars.