Carter was born on October 15, 1732, most likely in Christ Church Parish in Middlesex County, where he was christened on November 15, 1732. The son of Charles Carter, a wealthy planter and later a member of the House of Burgesses, and his first wife, Mary Walker Carter, and also a grandson of the land baron Robert “King” Carter, he was closely related to many of the colony’s leading families. Among his influential uncles were John Carter, secretary of the colony and a member of the governor’s Council, and Landon Carter, a noted planter, writer, and member of the House of Burgesses. On November 22, 1753, Carter married Elizabeth Chiswell, daughter of the land speculator and mine operator John Chiswell, of Hanover County. They had five sons and two daughters.