Following their marriage Joseph and Lucy lived on their farm in Tunbridge where three children Alvin, Hyrum and Sophronia were added to the family. They worked their farm for six years when they decided to rent the farm out and try their hand at store keeping. Stephen Mack was doing very well with his store in Tunbridge and there was not an existing store in Randolph which was on another branch of the White river in a valley just to the west some fifteen miles away. During their first year of business they learned about the demand for Ginseng which was an herb that was plentiful in New England soils and very profitable. So Joseph decided to collect and crystalize the ginseng roots as part of his business. He prepared a sufficient amount to ship to the Orient that it brought an offer of $3000 from a Mr. Stephens a resident of Tunbridge but Joseph determined to arrange for the sell of his crop on his own. He went to the New York harbor and contracted with the captain of a vessel headed to China to take care of the sell of his product. They anxiously waited for their ship to come in but when it arrived the captain only had a chest of tea for them. There would have been no question in their minds that something was terribly amiss. However, it was Stephen Mack who discovers what happened to their much needed profits. They were robbed of a reported sell of $4500 which they were unable to recover. They found themselves in a difficult position since so much of their stores inventory had been let out on credit or in exchange for ginseng roots taken in on barter. So when the Boston merchant that had provided the goods for their store called for payment they used the $1000 wedding gift towards the debt but that was not enough. To cover their financial obligations and not tarnish their family name they sold their farm basically for half price to pay off the final $800 of debt. This was a sad conclusion to what had been such a promising beginning. The Smiths had just crossed over the line from land owners to renters. After a short stay back in Tunbridge they accepted an offer to rent a little farm with a small house from Lucy's father Solomon who had recently acquired 100 acres in South Royalton and Sharon Townships. The farm was only some eight miles from Tunbridge so they would still be close to the Smith family as well as Lucy's parents and two brothers. Solomon and Lydia lived in a large house just to the north not even a mile away that sat next to the Old Turnpike Road that ran from Boston to Montreal and Daniel Mack and his family had a place just about a mile farther west. Joseph and Lucy had only been living in their small 20' by22' home just over a year when Joseph Jr. was born on Dec. 23, 1805. Because conditions for farming were not favorable the Smiths only stayed a couple years before moving on. They moved back to Tunbridge where another son Samuel was born and then moved to Royalton where Ephraim was born and died eleven days later and a year later in March of 1811 their son William was born. Later that year the family moved to Lebanon, New Hampshire.