Grimes was born in San Francisco, California. His uncle was actor Rudy Solari. Grimes' first major role was in the 1971 film Summer of '42, playing a teenager, based on author Herman Raucher, who has an affair with a beautiful older woman (Jennifer O'Neill). For the role, Grimes was nominated for a 1972 Golden Globe Award as Most Promising Newcomer, as well as the BAFTA Award as Most Promising Newcomer to Leading Film Roles. Grimes then starred in the sequel Class of '44 (1973), which followed his character to college.