Chambers is the son of Anne Cox Chambers, and the grandson of the newspaper publisher James M. Cox. He has two half-sisters from his mother's first marriage, Katharine Ann Johnson and Margaretta Johnson. He graduated from Bard College in 1982.
In 1982, Chambers married fellow Bard College alumna and actress Lauren Hamilton, who is the daughter of NASA scientist, mathematician and computer pioneer Margaret Hamilton and James Cox Hamilton, both of Cambridge, Massachusetts. They have a son, James Cox "Jim" Chambers Jr.