The building was started in the early 1500s. One of its owners, Sir Anthony Browne, acquired it in 1542 and also took control of Battle Abbey where he evicted the monks. As a result he was cursed that his line would perish by fire or water. Some 250 years later (in 1793) the house was destroyed by fire and at approximately the same time one of his heirs was drowned whilst trying to shoot waterfalls on the Rhine. Children of another heir were drowned in a boating accident in 1815, bringing the line to an end.