Rivers was projected to be an early first-round pick in the 2004 NFL Draft. However, despite Rivers's record of success in college and remarkable accuracy (72% completion percentage for his senior season), questions about his lack of arm strength and his unorthodox side-arm throwing motion were concerns for some NFL GMs. The pre-draft consensus was that Rivers could be selected by the Pittsburgh Steelers with the 11th pick. The San Diego Chargers coveted Eli Manning and wanted to select him with their first round pick, which was also the first overall pick of the draft. However, after Eli Manning indicated before the draft that he would not sign with them, Rivers was their first alternative because the Chargers head coach at the time, Marty Schottenheimer, had coached him at the Senior Bowl and he liked what he saw. The Chargers agreed to a draft day deal with the New York Giants. San Diego selected Manning first, then traded him to New York for Rivers, taken by the Giants fourth, plus draft picks later used by the Chargers on future Pro Bowlers Shawne Merriman and Nate Kaeding.
Rivers was one of 17 quarterbacks taken in the 2004 NFL Draft, along with Manning and the Pittsburgh Steelers' Ben Roethlisberger. All three would enjoy lengthy and successful careers with the teams that signed them and have been compared favorably to the 1983 NFL Draft, which included Hall of Fame quarterbacks John Elway, Jim Kelly, and Dan Marino. Of the three, Rivers has been the highest in pass completions, completion percentage, TD–INT ratio, passing yards, passer rating, and Pro Bowl honors. However, while Manning and Roethlisberger each have two championship titles, Rivers has not reached the Super Bowl during his career.
In August 2004, Rivers signed a six-year, $40.5 million contract with the San Diego Chargers that included $14.5 million in signing bonuses. However, due to a protracted contract negotiation with the San Diego Chargers, Rivers only reported to the team during the last week of training camp, and incumbent Drew Brees retained his starting job. Rivers began the season as the Chargers' third quarterback option, behind Doug Flutie, and ahead of the fourth quarterback on the depth chart, Cleo Lemon. Unfortunately for Rivers, Brees went on to have a then-career year and was named to the NFL Pro Bowl while winning the NFL Comeback Player of the Year Award. Rivers received very limited playing time, playing in only two games. He only threw passes in the second half of the last game of the 2004 season, a victory over the Kansas City Chiefs, by which time the Chargers had already clinched a home playoff spot and the AFC West division title. In the victory, he threw his first touchdown pass to fellow rookie Malcom Floyd. He was not on the active roster for San Diego's Divisional Round playoff loss to the New York Jets.