As a rookie, Jolly appeared in all 16 games as a defensive back for the 1980 Packers. As the fifth back in Green Bay's nickel defense, he had two interceptions and knocked down three passes as a rookie. Jolly was slated to be the Packers starting safety in 1981. During his first game as a starter, a preseason game against the Dallas Cowboys in August 1981, Jolly sustained an injury to his left knee which required surgery to repair ligaments. Jolly missed the entire 1981 season after the injury. He returned for the 1982 season but appeared in only seven games, none as a starter. In 1983, he appeared in 12 games for the Packers, none as a starter.