Seaham Hall was built in the 1790s by Sir Ralph Milbanke, 6th Baronet. In 1815 the poet Lord Byron married Anne Isabella Milbanke at Seaham Hall. The fruit of their marriage was Ada Lovelace, the mathematician and pioneer of computing.

Seaham Hall was one of the many properties acquired by Charles William Vane, 3rd Marquess of Londonderry through his second marriage to Lady Frances Anne Vane-Tempest in 1819. She was one of the greatest heiresses of the time. She stood to inherit nearly 65,000 acres (260 km2). They purchased the Seaham estate in 1821 from Sir Ralph Milbanke for £63,000 [1] and developed it into what is now the modern harbour town of Seaham. This town was designed to rival nearby Sunderland. The title Viscount Seaham was created as a courtesy title for the eldest son of the marriage, who became Earl Vane on his father's death; however, when the 4th Marquess of Londonderry died childless Earl Vane inherited the Londonderry titles and his eldest son took the courtesy title Viscount Castlereagh.

However, for much of his life the 5th Marquess lived at Plas Machynlleth, his wife's home in Montgomeryshire. The family did not spend much time at Seaham but used their Irish house - Mount Stewart, which was more impressive.

Benjamin Disraeli visited Seaham Hall in 1861.

Following the death of the 6th Marquess in 1915, his son the 7th Marquess put the hall at the disposal of the authorities to use as a hospital during the Great War,[2] and it subsequently continued in use as a general hospital before closing in 1978.