On May 6, 1856, Thomas Whaley began construction of his new home and proclaimed “My new house, when completed, will be the handsomest, most comfortable and convenient place in town or within 150 miles of here.” The two-story Greek Revival house was designed by Thomas Whaley and made from bricks created in Whaley’s very own brickyard on Conde Street. The house was the first of its kind and cost more than $10,000 when completed. Furnished with mahogany and rosewood furniture, Brussels carpets, and damask drapes, the Whaley House was overwhelmingly praised as “the first of its kind and finest home in Southern California.”

From 1869 through 1871, the Whaley House became the gathering place for the entire San Diego community. Besides being the Whaley family home, it was also San Diego’s first commercial theater, the county courthouse, and a general store. The home became such an integral part of the community that it was later the focal point in a battle between Old Town residents and New Town residents. The San Diego County government had been renting a portion of Whaley House for its offices and records. Whaley offered several times to sell the house to the county, but the offer was ignored. New Towners demanded that the county offices and records be moved to New Town. Despite threats of armed resistance from the Old Towners, the records were moved on March 31, 1871, effectively shifting the focus of the city to New Town.

Although the Whaley House was a communal treasure, it served primarily as the residential home for Thomas Whaley and his family. Unfortunately, tragedy continued to strike the Whaley family while they lived in the home leading to an increase in reported paranormal activity. In fact, the Whaley family told the San Diego Union shortly after moving in that they heard heavy footsteps in the house, which they believed to be the ghost of James “Yankee Jim” Robinson, who had earlier been hanged on the property for stealing a boat. Another tragic event, the violent suicide of Violet, one of the Whaley daughters, moved Thomas to build a new home in downtown San Diego and relocate his family.

In 1890, Thomas Whaley passed away in their new home on State Street, never to return to the original Whaley estate. Abandoned for close to 20 years, the Whaley House was neglected and in poor condition until Francis Whaley, their eldest son, decided to go back in 1909. After moving in, Francis set about restoring the property and entertaining tourists by playing his guitar and telling stories about the house’s background. In the years that followed, most members of the Whaley family returned to live and eventually die there.