Born in Los Angeles, MacArthur was the adopted son of playwright Charles MacArthur, and his wife, actress Helen Hayes. He grew up in Nyack, New York, along with the MacArthurs' biological daughter, Mary; she died of polio in 1949. He was educated at Allen-Stevenson School in New York, and later at the Solebury School in New Hope, Pennsylvania, where he starred in basketball, football and baseball.
MacArthur made his stage debut at Olney, Maryland in 1949, with a two-week stint in The Corn Is Green. His sister Mary was in the play and telephoned their mother to request that James go to Olney to be in it with her. The following summer, he repeated the role at Dennis, Massachusetts, and his theatrical career was underway.