The Castle Hill has an important role in the history and city structure of Esztergom. Grand Prince Géza located his quarters here and his son, St. Stephen was born here. The construction of the castle began under the reign of Géza and it was finished by Stephen. Esztergom became a royal seat and later the seat of the cardinals was also relocated here. During the centuries, Conrad III of Germany, Louis VII of France, Frederick I, Holy Roman Emperor were among the guests of the castle.

1198 was an important year in the history of Esztergom. This was the year when King Emeric gave the castle to the archdiocese and Esztergom started becoming an archbishopric seat.

During the Mongol invasion the castle successfully defended the city but it was badly damages in the war of succession after the dissolution of the Árpád dynasty in 1301. When the tutor of King Matthias, the humanist intellectual János Vitéz became archbishop, he started transforming the castle into a Renaissance archbishopric palace. His plan was finished only partly as after the Battle of Mohács it had to be used for defensive purposes again.

In 1543, the castle was seized by the Ottoman armies and reconquered in 1595 only to be taken again by the Ottomans for 10 years. It was only liberated in 1761 and used again by the archdiocese. The ruins of the palace were only excavated and restored in the first half of the 20th century and again in the 1960's.

The White Tower has been located in the southern end of the castle since the Árpád era. It was later used as a study and studiolo by János Vitéz which can be still seen today. The frescoes of the room depict the four virtues and according to some art historians, Temperance was painted by the young Sandro Botticelli.