Born on September 24, 1910, in Villa Rica, Georgia, Walker was the scion of a baseball family. His father, Ewart (the original "Dixie Walker"), was a pitcher for the Washington Senators (1909–12); an uncle, Ernie Walker, was an outfielder for the St. Louis Browns (1913–15); and his younger brother, Harry "the Hat" Walker, also an outfielder, played for four National League teams between 1940 and 1955 and managed the St. Louis Cardinals (1955), Pittsburgh Pirates (1965–67) and Houston Astros (1968–72). All four Walkers batted left-handed and threw right-handed.