After years of collecting on a small, personal scale, in 1891 Isabella inherited $1.75 million upon her father’s death and was able to begin collecting on a greatly expanded level. Upon purchasing Rembrandt’s Self-Portrait, Age 23 in 1896, Isabella and her husband Jack decided their ambitions as collectors required more space than their residence permitted, and first began to consider the idea of a museum.Isabella and Jack reached out to the architect Willard Sears, who had remodeled their house in Brookline. At first the couple considered expanding their current home, combining two houses on Beacon Street. However, as Isabella’s collection and ambitions continued to grow, Jack felt it would be more sensible to buy land and build a new building for the museum with apartments for themselves within it.  The Gardners loved Italy, and Isabella was especially passionate about Venice, where she and Jack would often stay at the Palazzo Barbaro on the Grand Canal. In the summer of 1897, Isabella and Jack traveled through Venice, Florence, and Rome to gather architectural fragments for their eventual gallery. They purchased columns, windows, and doorways to adorn every floor, as well as reliefs, balustrades, capitals, and statuary from the Roman, Byzantine, Gothic, and Renaissance periods.

In early 1899, Gardner purchased the land where the Museum now stands, and in June the pile driving began for the building’s foundation. Willard Sears, the hired architect, soon learned that Isabella was a challenging employer and meant to play a more active role than most clients. Throughout construction, she continually made changes, insisting that the workmen undo and redo their work, and Sears had to run interference between Isabella and his workers.