Worthy was born in Gastonia, North Carolina. His 21.5 points, 12.5 rebounds and 5.5 assists per game during his senior season at Ashbrook High led the team to the state championship game. Named both a Parade Magazine and McDonald's All-American, he was selected to play in the 1979 McDonald's All-American Game that featured future NBA Hall of Famers Isiah Thomas, Dominique Wilkins, and Ralph Sampson.
The Los Angeles Lakers had received the Cleveland Cavaliers' 1982 first-round draft pick in a 1979 exchange for Don Ford. The Cavaliers finished with the NBA's worst record in the 1981–82 season, leaving a coin toss to decide whether they or the worst record runner-up San Diego Clippers would get the first overall pick in the upcoming draft. The Lakers won the flip, the first and only time for a reigning league champion. They chose Worthy.