In 1958, the pacifist Rau and his political mentor, Gustav Heinemann, joined the Social Democratic Party of Germany (SPD), where he was active in the Wuppertal chapter. He served as deputy chairman of the SPD party of Wuppertal and was elected later on to the City Council (1964–1978), where he served as chairman of the SPD Group (1964–1967) and later as Mayor (1969–1970).
In 1958, Rau was elected for the first time as member of the Landtag (state parliament) of North Rhine-Westphalia. In 1967, he became chairman of the SPD fraction in the Landtag, and in 1970, he was Minister of Science and Education in the cabinet of Minister President Heinz Kühn. He soon gained a reputation as a reformer. As part of the mass education campaign of the 1970s, he founded five universities, each at different sites, in North Rhine-Westphalia and initiated Germany's first distance learning university at Hagen (modelled on the Open University).