Trafalgar Castle School was founded in 1874 as Ontario Ladies' College, making it one of the oldest independent schools in Canada. The castle was built by Nelson Gilbert Reynolds, Sheriff of Ontario County, as a private residence in 1859. Reynolds was named after Lord Horatio Nelson and named his castle Trafalgar in honour of the Battle of Trafalgar. The castle was the largest private dwelling in Canada until Casa Loma was built. It cost Reynolds $70,000 which was an exorbitant sum, a home at that time was built for $2,000 and a bank complete with vault could be built for $5,000 to $7,000. Reynolds was a colourful character and a gambling man. It was gambling losses that reportedly forced him to sell his beloved home to the Methodists in 1874 for the sum of $30,000.

The Methodists of that day were very interested in establishing higher learning institutions for young women. Rev. J. E. Sanderson convinced the Town of Whitby that this would be extremely beneficial to the economy of the town. James Holden, Esq. founder of the Dominion Bank, a local politician and businessman (with 5 daughters of his own) supported this idea. Sanderson and Holden were instrumental in raising the money and shares for the purchase of the castle and the establishment of Ontario Ladies' College.

Trafalgar Castle hosted its share of royalty and dignitaries through the years. Both as Reynolds' residence, then as Ontario Ladies' College and now as Trafalgar Castle School.