It is unclear what happened to the original inn, but the indications are that it burned during one of the periodic fires that raced through the old wooden structures in the crowded village. In 1824 a local cabinetmaker by the name of Nathan Tinker began construction of a brick edifice. It was probably used primarily as his residence.

By 1845, the whaling was near its economic peak and cultural nadir! Sag Harbor was filled to capacity with sailors, chandleries, the stench of rendering whale oil, and profligacy. Melville’s Quequeg, en route to Ahab’s Moby Dick, briefly stayed at Sag Harbor and concluded the heather way superior to that of the Christian! Whaling, three-masters and prosperity was subsumed by the gold rush of 1849-1850 in California, and Sag Harbor receded to a charming if impecunious backwater for over a century. As a business proposition, Nathan Tinker, Sag Harbor’s master cabinet maker, made a catastrophic mistake. He set to work adding onto his original brick structure for the purpose of providing mercantile space and housing to the whale trade. We may imagine immense stocks of hemp and tar, harpoons, sailcloth, barrels of hard tack and dried cod piled at the ready. A boarding house operated above and in the rear of the structure. It didn’t last long.

The whaling industry suffered a rapid decline after 1846. Ships had to travel much greater distances to find their catch. Steamships were replacing the wooden sailing vessels, and half the local fleet, over 20 ships were reportedly lost in the stormy winters of 1847 and 1848.

Captain William Freeman, whaler and ferry boat owner and Bridgehampton farmer Addison M. Youngs bought the building from Tinker’s heirs in 1876, built a porch, installed a bar and dining room, and named it The American Hotel or “The American House.”