In 1987 Vavi worked in a gold-mining territory of Klerksdorp and Orkney. He was a uranium plant clerk at Vaal Reefs mine, and joined the National Union of Mineworkers as an organizer. He was fired from AngloGold in 1987 following a massive miners’ strike which crippled the Chamber of Mines. He then joined COSATU as a volunteer.

In 1988, he became COSATU’s regional secretary for the Western Transvaal. In 1992 he took up the position of national organizing secretary. He served as COSATU’s deputy general secretary from 1993 to 1999.

In 1999 he succeeded Mbhazima Shilowa (formerly the Gauteng premier, and later one of the leaders of COPE) as general secretary of COSATU.

As general secretary he has expressed disappointment with political progress in Zimbabwe, blaming mismanagement by government, an ill-conceived structural adjustment programme, and factors beyond its control like sanctions against neighbouring South Africa during apartheid years. He led two fact-finding missions, in October 2004 and February 2005, to assess conditions for a fair poll during the planned Zimbabwe parliamentary elections of 2005. The members of the mission were deported during the first mission, and were turned back at Harare airport at the start of the second.

Talking of the alliance which governs South Africa between the ANC, COSATU and the SACP after a failed trip to see the Zimbabwe Congress of Trade Unions (ZCTU) in 2004, he is critical of the ANC’s approach to Zimbabwe, privatisation, and corruption in government.

In May 2010 the ANC threatened to charge Vavi with “ill-discipline”.

Vavi was suspended from the Cosatu secretary-general position in 2013 after a rape accusation.

Vavi has spoken out against the expulsion of NUMSA, the metalworkers union, which used to be the largest union in the COSATU federation. This is one of the major factors contributing to Vavi’s expulsion from COSATU. On 30 March 2015 the Central Executive Committee (CEC) voted to expel Vavi. The vote was 31–1 in favour of expelling Vavi.