Bryan was born and raised in the area of Bristol. He came through his local team’s youth levels; signing his first professional contract in 2011, agreeing terms to be a Bristol City player for at least two years.

On 24 November 2011, Bryan joined Conference National side Bath City on a two-month loan deal, effectively lasting until January. He scored in a 3–1 win over A.F.C. Telford United.

He made his professional debut for Bristol on 6 March 2012, in a 3–2 win over Leicester City at Ashton Gate. In March 2013, he joined Plymouth Argyle on loan until the end of the season. Bryan made his debut for Plymouth the same day against Bradford City and scored his first Football League goal at Chesterfield in April.

Bryan was selected by the Scotland national under-21 football team in October 2012, but could not play due to a concussion he received playing for Bristol City against Leeds United.

He returned to Bristol City at the end of the season having played on the left side of midfield in ten consecutive games for Argyle. Bryan scored his first goal for City in the Bristol derby with Rovers in September, and his first league goal for the club came in November against Crawley Town. In June 2015 he signed a contract extension keeping him at Bristol City until 2019. The 2016–17 campaign saw Joe Bryan become statistically the best left back in the division.

On 9 August 2018, Bryan signed for newly promoted Premier League club Fulham for a transfer fee reported to be in the region of £6 million. He scored his first goal for Fulham in an EFL Cup tie against Millwall on 25 September 2018. On 4 August 2020, Bryan scored twice in extra-time in the Championship play-off Final at Wembley Stadium to send Fulham back up to the Premier League after a one-year absence, with the first one being a long-range direct free kick.