Born in Chicago, Illinois, Jones moved to Alabama, where he learned to play several instruments, including saxophone, piano, and drums. He worked as a drummer and tap-dancer at carnival shows until joining Walter Page’s band, the Blue Devils in Oklahoma City in the late 1920s. He recorded with trumpeter Lloyd Hunter’s Serenaders in 1931, and later joined pianist Count Basie’s band in 1934. Jones, Basie, guitarist Freddie Green and bassist Walter Page were sometimes billed as an “All-American Rhythm section,” an ideal team. Jones took a brief break for two years when he was in the military, but he remained with Basie until 1948. He participated in the Jazz at the Philharmonic concert series.

In 1979, Jones was inducted into the Alabama Jazz Hall of Fame for his contribution to the Birmingham, Alabama musical heritage. Jones was the 1985 recipient of an American Jazz Masters fellowship awarded by the National Endowment for the Arts.

His autobiography (as told to Albert Murray), entitled Rifftide: The Life and Opinions of Papa Jo Jones and based on conversations between Jones and novelist Murray from 1977 to before Jones’ death in 1985, was posthumously published in 2011 by the University of Minnesota Press.