Like his older brother Ben, Jarvis started his career on the books of Millwall, but was released by the club. With the help of a member of the Millwall coaching staff, however, he was taken on by Kent-based club Gillingham as a trainee. On 4 November 2003, at the age of 17, he made his Football League debut in a match against Sunderland when a number of senior players were missing due to influenza. He came on as a 76th-minute substitute for Richard Rose in a match which his team lost 3–1. He next played for the first team in an FA Cup defeat to Burnley on 24 January 2004, and went on to feature regularly in the team in March and April. In total he made two starts and eight substitute appearances during the 2003–04 season, but also continued to play for the youth team, helping them reach the last sixteen of the FA Youth Cup. Shortly before the end of the season he signed his first professional contract with the club when manager Andy Hessenthaler gave him a three-year contract.

In the 2004–05 season Jarvis became a regular in the Gillingham first team, playing in 30 Football League matches, although he was out of action for five weeks in January and February after undergoing an operation on a hernia. He also scored his first goal for the club in a win over Wolverhampton Wanderers on 30 October 2004. Gillingham, however, were relegated from the Championship at the end of the season. In the 2005–06 season he was again a regular, and scored seven goals, his best season tally to date.

Jarvis rejected a further improved contract offer by Gillingham, to join Wolverhampton Wanderers of the Championship in June 2007, signing a two-year contract with the option to extend his stay by a further year. The clubs agreed a transfer fee but the exact amount was not disclosed.

Jarvis suffered hip and groin injuries during pre-season training and was unable to take any further part in training until September. He finally made his debut for his new club on 20 October 2007, when he came on as an 88th-minute substitute in a 2–0 home victory over Charlton Athletic. After regaining full fitness, he became a first-choice player at Molineux, making a further 27 appearances during the 2007–08 season. He scored his first goal for the club in a 1–1 draw with Leicester City in December 2007, in a season where the club missed out on a play-off place by virtue of goal difference.

After a strong start to the following season for both club and player, Jarvis injured his hamstring in the defeat to Reading on 30 September 2008 and was expected to be out of action for approximately six weeks. He made his return in the 1–0 defeat to Queens Park Rangers on 6 December, and remained a first choice player as the club won promotion to the Premier League as champions.

He made his Premier League debut against West Ham United in August 2009 and played regularly during the 2009–10 season, scoring three times to help them achieve top flight survival. In September 2010, he signed a new contract which would have kept him at the club until the summer of 2015, and was once again a regular in the Wolves team during the 2010–11 season, scoring four Premier League goals, his highest tally for four seasons. On 19 May 2011, Jarvis was awarded both Supporters’ Player of the Year award and Players’ Player of the Year awards.

Jarvis was called up to the England national team in March 2011 for matches against Wales (a Euro 2012 qualifier) and Ghana (an international friendly). After not being selected for the matchday squad against Wales, he made his debut as a substitute against Ghana at Wembley Stadium on 29 March, coming on for Jack Wilshere in the 70th minute of a 1–1 draw. In doing so, he became the first Wolves player for England since Steve Bull in 1990.

Jarvis submitted a transfer request on 11 August 2012, after media reports linked West Ham United with the player. Wolves rejected several bids for the player, without naming the club involved. On 23 August, Wolves accepted an offer from West Ham of a reported, initial £7.5 million, rising to £10.75 million.

On 24 August 2012, Jarvis signed a five-year contract, with the option for an additional year, for a transfer fee which was a club record but was officially undisclosed. Jarvis made his debut for West Ham on 25 August in a 3–0 away defeat to Swansea City. He scored his first goal for the club on 1 October, opening the scoring in a 2–1 win away to Queens Park Rangers at Loftus Road.

Jarvis has been married to his wife Sarah since June 2011. In January 2013, he was interviewed for, and appeared on the cover of, gay lifestyle magazine Attitude to discuss homosexuality in football.

On 1 September 2015, Jarvis signed on a season-long loan for Norwich City. Eleven days later, he scored his first goal for the club against AFC Bournemouth in a 3–1 win. On 24 December 2015, Jarvis signed a permanent contract with Norwich, to take effect in the January transfer window, for a reported fee of £2.5 million. Jarvis started 12 games for the Canaries in the 2015–16 season, coming off the bench as a substitute another eight times. Nearing the end of the season Jarvis suffered a knee injury, which along with hamstring and hip injuries, would cause him to miss the entire 2016–17 season, with the exception of one appearance for the club’s under-23 team in the EFL Trophy.

On 1 January 2019, Jarvis joined League One club Walsall on loan until the end of the 2018–19 season. He was released by Norwich at the end of the season.

Jarvis signed a deal with Woking on 28 February 2020. On 29 February he scored his first goal for the club on his debut, in a league game against Maidenhead United. On 7 July 2020, the club announced that Jarvis had extended his deal until the end of the 2020–21 season.