The inn known as "The Anchor" was built during the first reign of Edward IV (1461-1470) in the year 1465. The Anchor appears on a map of 1598 when it was referred to as Broome Place, being the manor house of Broome and likely dates from the 15th century. Broome Manor was purchased around 1584 by Sir Thomas Sackville.

Hartfield is on the northern edge of the Ashdown Forest, which many know as the setting for AA Milne's Winnie-the-Pooh books and is located a stone's throw from Bolebroke Castle, another of the Sackville estate properties. By the late 18th century and certainly by 1777, the manor had declined and become a poorhouse or workhouse. A reference to a workhouse appears in the 1821 census which lists 54 inmates.

In 1834 the poorhouse became part of the East Grinstead Union incorporating the parishes of East Grinstead, West Hoathley, Worth, Crawley, Hartfield, Withyham and Lingfield. Hartfield took the girls - families were therefore split up. In the 1841 census, three inmates are male of which only one is a mature man, the other two being infants. The remaining occupants are all girls, the oldest being 19. The inmates were given work-related tasks but many were placed in families as domestic servants. Penalties for misbehaving or running away included solitary confinement and bread and water rations.

By the 1851 census, the inmates' occupations are listed as scholars, presupposing that the workhouse had become a school. By 1861, however, the workhouse was already being run down and in this year was purchased by one William Garret for the purpose of selling beer, although in the 1891 census it was still referred to as 'The Old Workhouse'. The name 'Anchor Inn' first appears on a public document in 1891, by which time Nicholas' wife, Mary Garrett, was the landlady.