Following his military service, he started actively working for the Koç Group in 1958, joining the Otokoç Company in Ankara. In 1960, he transferred to Koç Ticaret, which represented the Koç Group in Ankara. Following the relocation of the Koç Holding headquarters from Ankara to Istanbul in 1964, which was established a year earlier, he became general coordinator of Koç Holding A.Ş. and moved to Istanbul.

Thereafter in 1970, he became chairman of the executive committee and in 1975 deputy chairman of the board of management. Rahmi Koç was appointed chairman of the Koç Holding management committee in 1980. His father Vehbi Koç transferred his chairmanship to his son on 30 March 1984. Rahmi Koç retired from this duty on 4 April 2003, transferring his chair to his eldest son Mustafa Koç. Rahmi Koç is currently Honorary Chairman of the Koç Holding.

In 2013, he bought the Merrill-Stevens Drydock & Repair Co., Florida’s oldest boatyard, and renamed it RMK Merrill-Stevens. He is the owner of a 171-foot yacht, the Nazenin V.

Rahmi Koç has three sons: Mustafa Vehbi (1960–2016), Ömer Mehmet (1962) and Ali Yıldırım (1967) from his previous marriage to Çiğdem Simavi. His son Mustafa died on 21 January 2016, of a heart attack, at the age of 55.