Vajpayee retired from active politics after the 2004 defeat, putting Advani to the forefront of the BJP. Advani became Leader of the Opposition in the Lok Sabha from 2004 to 2009. During this period, Advani had to deal with rebellion from within the party. His two close associates, Uma Bharati, and Madan Lal Khurana, and longtime rival Murali Manohar Joshi publicly spoke out against him. In June 2005, he drew much criticism when he, while on a visit to the Jinnah Mausoleum at Karachi – his town of birth, endorsed Mohammad Ali Jinnah and described him a "secular" leader. This did not sit well with the RSS either and Advani was forced to relinquish his post as BJP president. However, he withdrew the resignation a few days later.
The relationship between Advani and the RSS reached a low point when the latter's chief K. S. Sudarshan opined that both Advani and Vajpayee give way to new leaders. At the Silver Jubilee celebrations of the BJP in Mumbai in December 2005, Advani stepped down as party president and Rajnath Singh, a relatively junior politician from the state of Uttar Pradesh was elected in his place. In March 2006, following a bomb blast at a Hindu shrine at Varanasi, Advani undertook a "Bharat Suraksha Yatra" (Sojourn for National Security), to highlight the alleged failure of the ruling United Progressive Alliance in combating terrorism.