In 1924 a meeting was held in Penrith by Arthur Judges who saw the benefits of keeping items of Australian history alive for future generations.

At the meeting, a resolution was passed to form the Nepean District Historical Society. Another meeting was reported to elect the office bearers, but after that, there is no further knowledge of this group.
The society was officially re-established in 1947. It continued to operate from the private homes of members until 1971 when it was granted use of the old ‘Arms of Australia Inn’ building that had been left in disrepair.

Five years later, after extensive restoration and hard work, the society relocated to the inn and it has been the headquarters of the society ever since.

The inn was once a staging post for travellers making the trip from Sydney over the Blue Mountains to Bathurst and the goldfields. Built in two sections, the first in 1826, it is one of the oldest buildings in Penrith, if not the oldest.