Wolterton was built by Horatio Walpole, the brother of the statesman Sir Robert Walpole, in 1727–41. Lady Dorothy Nevill, in Mannington and the Walpoles (1894), writes:
There is a family ghost at Wolterton, which at intervals is seen by old servants about the place. A white lady is said to be in the habit of appearing whenever some calamity is about to threaten our family. Some little time before my brother, the late Lord Orford, died, in 1894, I well recollect his saying to me, ‘I hear from Norfolk that the white lady has been seen again. It is you or I this time, Dolly, for we are the only ones left[.]’ The white lady in question is supposed to be one of the Scamler family, who were the possessors of Wolterton before my ancestor built the present mansion. There used to be some story that one of the Lords Orford unearthed the old tombstones of the Scamlers in the ruined church in Wolterton Park, and that this act of sacrilege was the cause of the poor lady’s spirit being so disturbed … In old days the Walpoles used to be driven in their hearse three times around this ruined church before being laid to rest in the family vault [at Wickmere]. Certainly Lady Walpole of Wolterton (Pierre Lombard’s daughter) was buried with this ceremonial.
This custom was said by later writers to have been devised in order to placate her. Lady Dorothy Nevill, however, reports discovering that no such act of sacrilege was ever perpetrated, ‘so it must be for some other reason that the ghostly dame lingers about Wolterton’.
The last burial in the churchyard at Wolterton took place in 1747, and Fadens’ map of Norfolk in 1797 records it as a ruin. The ceremony connected with it perhaps inspired the tradition of a phantom hearse, ‘with steeds, plumes, and attendants, according to some all headless’, said to appear to Walpoles as a death-warning at both Wolterton and Raynthorpe Hall.
Lady Dorothy Nevill says also that the Lady Walpole née Lombard mentioned was herself believed to haunt the house:
In the drawing-room is a full-length portrait of Ambassador Horace Walpole. This gentleman formed part of a large picture comprising himself and wife and seven or eight children, some of which are represented as angels, apparently having died as babies. My father cut this picture up and gave the portraits to different members of the family, whose descendants they are. The unhappy wife, Miss Lombard, is said to haunt Wolterton seeking for her divided relatives.