Plimpton was married twice. His first wife, whom he married in 1968 and divorced in 1988, was Freddy Medora Espy, a photographer's assistant. She was the daughter of writers Willard R. Espy and Hilda S. Cole, who had, earlier in her career, been a publicity agent for Kate Smith and Fred Waring. They had two children: Medora Ames Plimpton and Taylor Ames Plimpton, who has published a memoir entitled Notes from the Night: A Life After Dark.
At Harvard, Plimpton was a classmate and close personal friend of Robert F. Kennedy. Plimpton, along with former decathlete Rafer Johnson and American football star Rosey Grier, was credited with helping wrestle Sirhan Sirhan to the floor when Kennedy was assassinated following his victory in the 1968 California Democratic primary at the former Ambassador Hotel in Los Angeles, California. Kennedy died the next day at Good Samaritan Hospital.