Advertisement “The architecture of this town affects you beyond words,” he said, adding that Newburyport’s trails, museums, and beaches are well worth the trip. Ahead, discover some of Jones’s picks for where to go and what to do in Newburyport. Take in the architecture. Newburyport’s brick downtown buildings and historic federal-style architecture are what makes the city special, Jones said. “We do have the Colonial period, we have the Federalist period, we have the Victorian period, we have almost good representation of every architectural period,” Jones said. “But clearly the Federalist period is something we hold our hat on.”