Ziani began his football career in 1995 with the junior side at Racing Club de Paris at the age of 13.

In 1998, he was spotted by scouts from Troyes, he was subsequently brought into the youth academy from RC Paris. Whilst playing for the youth team there was a significant amount of keen admirers interested in him. A number of clubs such as Lens and Nantes were very keen on him. In 2001, having spent three seasons playing with the youth team and because of the level of interest, the president of Troyes offered Ziani a five-year deal, which in turn was his first professional contract. With Ziani agreeing to a five-year deal, he was immediately promoted to the senior squad by Alain Perrin at the age of 17 and was assigned the number 17 shirt. He made his professional debut on 8 December 2001 against SC Bastia coming on as a substitute, replacing David Hamed in the seventy-second minute of the game. After making a few substitute appearances, he made his first start on 3 March 2002 in a 1–0 loss to Lille playing the full length of the game. He played a total of 8 games in the 2001–02 season. He became more of a regular in the 2002–03 season as he participated in a total of 24 games. After subsequently spending two seasons playing for Troyes in Ligue 1, the club was relegated to Ligue 2 at the end of the 2002–03 season. Whilst in Ligue 2, during the 2003–04 season, Ziani played 28 games in total and scored his first professional goal against FC Rouen, in the 9th minute with the score ending 3–0. He spent the following season mostly on the bench for Troyes and with interest from other clubs being shown, Troyes had decided to part ways with Ziani. Turkish club Rivaspor had made a bid but could not negotiate a contract with Ziani, as Ziani had put in a condition which Rivaspor rejected. Ziani revealed the condition merely as having Madjid Bougherra have a trial with the club to display his qualities as they had both travelled to Turkey and Ziani would have liked a compatriot with him at the club as he was only 19 at the time. When Rivaspor rejected the condition, both Ziani and Bougherra left Turkey together.

Ziani had stunned many people in 2003 in his choice to represent Algeria instead of France.

Ziani made his debut for the Algerian national team in 2003 when he was called up by Belgian caretaker Georges Leekens for a friendly against Belgium. He was a member of the Algerian 2004 African Nations Cup team, who finished second in their group in the first round of competition before being defeated by Morocco in the quarter-finals. He was also named in the Team of the Tournament at the 2004 African Nations Cup. Karim Ziani scored his first goal for Algeria against Gambia with a penalty in a 2008 African Nations Cup qualifier, Algeria won 1–0. Ziani then added two more goals to his tally in a 2010 World Cup against Liberia. Algeria won the game 3–0. Ziani was also the artist of the recent Algerian team performances during the final phase of the 2010 FIFA World Cup qualification together with 2010 African Cup of Nations.

The following summer, Ziani saw increasing attention from other French clubs, namely Olympique de Marseille. This interest culminated on 29 June 2007, when news broke that Ziani had finalized his move to Marseille for a transfer fee of $11 million. Karim scored his first goal with Marseille in a match against Valenciennes. He became a familiar face in Eric Gerets’ starting eleven.

On 7 July 2009, it was announced that Ziani had signed a four-year contract with Bundesliga champions VfL Wolfsburg.

During December 2010 and January 2011, a number of clubs showed interest in signing Ziani, these were Newcastle United, AS Monaco, FC Lorient, VfB Stuttgart, Everton, and Birmingham City. On 14 January 2011, it was reported that Ziani had moved to Turkish high-flyers Kayserispor on an initial six-month loan deal, with an option to sign him permanently at the end of the season.

On 8 July 2011, Wolfsburg and Ziani reached a mutual agreement to terminate his contract.

On 20 July 2011, Ziani signed a three-year contract with Qatari club El Jaish. On 18 September 2009, Ziani made his Qatar Stars League debut for El Jaish as a starter in a league game against Al Khor. Although he failed to find the net in his first season, he finished the 2011–12 season as the top assistant with 13 assists.

In summer 2019, Ziani ended his 18-year career at the age of 36.