Born on September 24, 1910, in Villa Rica, Georgia, Walker was the scion of a baseball family. His father, Ewart (the original “Dixie Walker”), was a pitcher for the Washington Senators (1909–12); an uncle, Ernie Walker, was an outfielder for the St. Louis Browns (1913–15); and his younger brother, Harry “the Hat” Walker, also an outfielder, played for four National League teams between 1940 and 1955 and managed the St. Louis Cardinals (1955), Pittsburgh Pirates (1965–67) and Houston Astros (1968–72). All four Walkers batted left-handed and threw right-handed.

Walker originally entered the major leagues with the New York Yankees, and was considered the heir to Babe Ruth as the team’s left fielder after playing with the Yankees in 1931, and again from 1933 to 1936.

In 1936 Walker married Estelle Shea. They were the parents of daughters Mary Ann and Susan, and sons Stephen, Fred Jr., and Sean.

After stints with the Chicago White Sox and Detroit Tigers, Walker blossomed into a star with the Brooklyn Dodgers, with whom he played from 1939 to 1947. He was a five time All Star, being selected in every year from 1943 to 1947. In addition, he was the National League’s batting champion in 1944, with his average of .357 besting runner up Stan Musial’s .347. In addition, Walker was the 1945 National League runs batted in champion, with his total of 124 topping Boston Braves outfielder Tommy Holmes, with 117. After the 1946 season, Walker became the first National League “player representative” recognized by major league owners. The post was created to stave off the formation of a players’ union in the wake of the short-lived American Baseball Guild movement earlier that year. Johnny Murphy, the Yankees’ stalwart relief pitcher, became the American League’s first player representative.

After the 1947 season, Walker was traded to the Pittsburgh Pirates, for whom he played two seasons before retiring in 1949.

Walker vocally opposed the participation of black baseball players regardless of their skill. He suggested he would not play for the Dodgers if a black baseball player were permitted on the team. He reportedly initiated a player petition within the Dodgers in 1947, opposing Jackie Robinson joining the team, and he wrote a letter to Dodgers owner Branch Rickey asking to be traded.

Walker coached with the St. Louis Cardinals in 1953, but left partway through the season to manage the Cardinals’ Houston team in the Texas League. He managed Houston through 1954, after which he managed in the International League, first with the Rochester Red Wings (1955–1956), where he won back-to-back Governors’ Cup championships, and then with the Toronto Maple Leafs (1957–1959).

The Milwaukee Braves made Walker a scout in 1960, and he worked in this position until 1963, when he joined the team’s coaching staff for three years. When the Braves relocated to Atlanta in 1966, Walker was their chief scout for the Southeastern United States.

In 1968, Walker rejoined the Dodgers as hitting coach, and he held this position until 1974. From 1974 to 1976 he was a coach for the Dodgers’ minor league system.

In a 1981 interview, Walker explained that his trade request was not due to Robinson, but because Walker had become a scapegoat for opposition within the team. In his 2002 book, The Era, 1947–1957, author Roger Kahn wrote that Walker admitted to starting the Dodgers’ player petition in 1947, in which the signers opposed the integration of baseball. In an interview with Kahn, Walker stated, “I organized that petition in 1947, not because I had anything against Robinson personally or against Negroes generally. I had a wholesale business in Birmingham and people told me I’d lose my business if I played ball with a black man.” According to Kahn, Walker referred to the petition as “the stupidest thing he’d ever done” and asked Kahn that, if he had the opportunity, he would write that Walker was sorry and apologized for his actions.

Walker died of colon cancer in Birmingham on May 17, 1982, and was buried at Elmwood Cemetery in Birmingham.

In his 2014 book, Rickey & Robinson: The True, Untold Story of the Integration of Baseball, Kahn also wrote that, in May 1947, Walker — the NL’s player representative — proposed the idea of a league-wide players’ strike in an attempt to end Robinson’s MLB career. Although the St. Louis Cardinals reportedly were in favor of the idea, the quick intervention of their owner, Sam Breadon, and National League president Ford Frick, immediately reported by Stanley Woodward in the New York Herald Tribune, destroyed the strike movement.