He moved abroad to join English club Charlton Athletic in a DKK 8 million transfer deal in June 2004. In his first season with the London club, he only made two first team appearances. Due to an injury sustained by first choice goalkeeper Dean Kiely, Andersen started the 2005–06 season in the team. Charlton's poor mid-season form saw him dropped in favour of Kiely at first, then Thomas Myhre, who established himself as first choice as Charlton turned their season around. Andersen was reinstated into the lineup for the last two games of the season, when Myhre suffered an injury. In the 2006 summer transfer window, Andersen looked ready to join Leeds United on a season-long loan deal, but as Charlton insisted on a clause to allow the club to recall Andersen at any time during the season, the transfer fell through at the last moment. In the first half of the 2006–07 season, he saw the loaned-in Scott Carson become Charlton's starting goalkeeper.