He was younger brother of John Charles Spencer, 3rd Earl Spencer , and succeeded his brother who was politically active in the House of Commons as Lord Althorp (a member of Lord Grey's cabinet).

By his 1st marriage, the 4th Earl Spencer was father of the 5th Earl Spencer (better known in his day as the Red Earl); and by his second marriage, he was father of the 6th Earl Spencer. This earl in turn was father of Albert Edward John, 7th Earl Spencer, who was father of Edward John, 8th Earl Spencer (d. 1992), father of Diana, Princess of Wales. The 4th Earl is thus a direct ancestor of Diana.