Harbaugh attended college and played varsity football as a defensive back at Miami University, where he graduated in 1984.

He was first hired in the NFL in 1998 by the Philadelphia Eagles’ then head coach Ray Rhodes, and was one of four assistant coaches retained by new head coach Andy Reid in 1999. As such, he is in the Sid Gillman coaching tree. In 2004, he was mentioned as a possible candidate to replace Gary Darnell as the head football coach at Western Michigan, where he had earned a master’s degree and was an assistant football coach from 1984–1987.

In 2007, after serving as Eagles’ special-teams coach for nine years, he became their defensive-backs coach. This fulfilled his request to head coach Reid and improved his chances of landing a head coaching job since executives at that time viewed special teams coaches as unqualified to move up to head coach.

On January 19, 2008, Harbaugh was appointed the third-ever head coach of the Baltimore Ravens after Jason Garrett, the team’s first choice, decided to stay with the Dallas Cowboys after receiving a raise and a promotion to assistant head coach. He was not considered one of the favorites for the position because he had no head coaching experience at any level and had never been an offensive or defensive coordinator in the NFL. He impressed team owner Steve Bisciotti and Vice President of Player Personnel/General Manager Ozzie Newsome. New England Patriots head coach Bill Belichick also recommended Harbaugh to Bisciotti by phone during the interview process.

On January 23, 2008, Harbaugh hired longtime NFL offensive coach (and former head coach) Cam Cameron as offensive coordinator. (Cameron had previously hired Harbaugh as an assistant at Indiana.) Cameron was also quarterbacks coach for John’s brother, Jim, during their time at Michigan. On September 7, 2008, in his debut as a head coach, John and his Ravens beat the Cincinnati Bengals.

On January 26, 2009, he named Greg Mattison the defensive coordinator for the Ravens, replacing Rex Ryan who had left to take his first head coaching job (with the New York Jets). Mattison had served as linebacker coach and defensive coordinator for Harbaugh’s father, Jack, at Western Michigan University from 1981–86, when Harbaugh was a graduate assistant and assistant coach for his father.

In his second season as Ravens’ head coach, he once again led the team to the playoffs with a 9–7 record during the regular season and improved his playoff record to 3–1 with an upset victory over the New England Patriots in the AFC wild card round on January 10, 2010 before losing in the AFC divisional game to the Indianapolis Colts. He once again took the Ravens to the playoffs in 2010, beating the Kansas City Chiefs in the wild card round on January 9, 2011, before losing to the Pittsburgh Steelers in the divisional round 31–24 on January 15 after starting the second half with a 14-point lead.

Harbaugh signed a three-year extension on February 14, 2011 that kept him under contract through 2014. The Ravens finished 2011 12–4, winning the AFC North division and sweeping the Steelers home and away before losing the AFC Championship Game to the New England Patriots after Lee Evans had a potential late game-winning pass knocked out of his hands by Patriots defensive back Sterling Moore and kicker Billy Cundiff flubbed a potential game-tying field goal. Neither Evans nor Cundiff made the 53-man 2012 roster.

The 2012 Baltimore Ravens again met the Patriots in the AFC championship game (on January 20, 2013), got their revenge with a 28–13 victory (coming from behind with a 13–7 second half), and was the first time Tom Brady and Bill Belichick lost a home game after leading at halftime, giving John the opportunity to face brother Jim and the 49ers in Super Bowl XLVII on February 3, 2013. Many have pegged Super Bowl XLVII as the “Harbowl”. The Ravens were victorious, defeating the 49ers 34–31. Following the victory, John gave his entire staff replica Lombardi trophies to commemorate the victory.

In 2012, Harbaugh was awarded the third-highest honor within the Department of the Army Civilian Awards, the Outstanding Civilian Service Award, for substantial contributions to the U.S. Army community while serving as the Baltimore Ravens Coach.

He was selected to be inducted into Miami University’s “Cradle of Coaches” in 2013.

On September 5, 2013, an hour before the Ravens played in the NFL regular season’s opening game, it was reported that Harbaugh had signed a four-year contract extension in a deal that was reached “months ago.”

In each of Harbaugh’s first four seasons and again in 2014, every AFC Champion defeated the Ravens in the playoffs (although only the 2008 Pittsburgh Steelers and 2014 New England Patriots were able to actually win the Super Bowl).

In the 2014 AFC Wild Card round of the NFL playoffs, Harbaugh’s Ravens beat the Pittsburgh Steelers in Heinz Field in a dominant 30–17 victory, which was the Ravens’ first playoff victory against the Steelers in the history of the franchise. However, the next week, the Ravens lost 31–35 in the AFC Divisional round to the New England Patriots after the Ravens were unable to hold two separate 14-point leads. After the game, Harbaugh complained about the Patriots’ uncommon but legal tactics of declaring receivers eligible and ineligible, saying “It was clearly deception.”

In 2015, Harbaugh had his first losing season with the Ravens. The Ravens lost many close games and key players like Joe Flacco, Justin Forsett, Steve Smith Sr., Eugene Monroe, and Terrell Suggs all suffered season-ending injuries. They finished third in the AFC North with a 5–11 record.

On August 28, 2017, Harbaugh signed a one-year contract extension, keeping him under contract through the 2019 season.

On January 24, 2019, Harbaugh signed a four year contract extension, keeping him under contract through the 2022 season.

During the season, Harbaugh lead the Ravens to a 14–2 record in the regular season and secured the number 1 seed in the AFC playoffs. In the divisional playoff game against the Tennessee Titans, the Ravens lost the game 28–12. For his work during the 2019 season, on February 1st, 2020, Harbaugh was honored as the Associated Press NFL Coach of the Year Award.