After two seasons in the Indians’ farm system with the AAA Portland Beavers, Agee received a September call-up to Cleveland in 1962. With the Indians already behind 11-1 to the Minnesota Twins, Agee made his major league debut on September 14 at Metropolitan Stadium pinch-hitting for pitcher Bill Dailey in the ninth inning. He received September call-ups to the majors the following two seasons as well, playing a total of 31 games with the Indians in which he batted .170 with one home run and five runs batted in. Following the 1964 season, he was dealt to the Chicago White Sox with Tommy John as part of a three team blockbuster trade between the Indians, White Sox and Kansas City Athletics that returned All-Star Rocky Colavito to the Indians.

Agee batted just .226 with the Pacific Coast League’s Indianapolis Indians, and .158 in ten games with the White Sox in 1965. After earning the starting center fielder job in spring training 1966, he hit a two-run home run in the season opener, and was batting .264 with nine home runs and 38 RBIs to be named the White Sox’s sole representative at the 1966 Major League Baseball All-Star Game. He ended the season with a .273 batting average, 22 home runs and 86 RBIs to earn American League Rookie of the Year honors, while his defense in center field earned him a Gold Glove. Although he was technically in his 5th major league season, MLB’s classification of a rookie is determined by plate appearances and time on a major league roster. Agee’s September call-ups had been so brief and his playing time so scarce that he was still eligible for the award.

Agee was batting .247 with ten home runs and 35 RBIs to earn his second consecutive All-Star selection in 1967. His production fell off considerably in the second half of the season (he hit only four home runs after the All-Star break), and ended the season batting .234 with 52 RBIs. Though they finished the season in fourth place, Chicago finished only three games back of the first place Boston Red Sox, and battled Boston, the Detroit Tigers and Minnesota Twins until the final week of the season. On a team loaded with pitching and short on offense (no regular batted over .241), the team’s lack of offense possibly cost the White Sox the American League pennant.

Agee batted .357 with two home runs and four RBIs in the Mets’ three-game sweep of the Atlanta Braves in the 1969 National League Championship Series. The Mets were heavy underdogs heading into the World Series against the Baltimore Orioles. In Game 3 (the first World Series home game in Mets history), with the series tied 1–1, Agee had what Sports Illustrated called the greatest single performance by a center fielder in World Series history. In the first inning, Agee hit a leadoff home run off Jim Palmer for what would eventually be the game-winning hit and RBI, as the Mets shut out the Orioles, 5–0. In the same game, Agee also made two catches that potentially saved five runs. The first catch came in the fourth inning with Gary Gentry pitching and two outs and runners on first and third. Agee, playing the left-handed hitting Elrod Hendricks to pull, made a backhanded catch near the base of the wall in left centerfield. The second catch came in the seventh inning with Nolan Ryan relieving Gentry; the bases were loaded with two outs, and Agee made a headfirst dive in right centerfield on a ball hit by Paul Blair.

Chronic knee injuries hampered Agee in 1971 and 1972, though he still batted .285 and tied for the Mets lead with 14 home runs in 1971. In 1972, he finished second on the Mets with 47 RBIs despite batting only .227.

Agee was traded from the Mets to the Houston Astros for Rich Chiles and Buddy Harris at the Winter Meetings on November 27, 1972. He faced the Mets for the first time in his career on April 24, and went two for three with a walk and a run scored in the Astros’ 4-2 victory. He was batting .235 with eight home runs and 15 RBIs when the Astros dealt him to the St. Louis Cardinals on August 18, who were in a battle with the Mets and Pittsburgh Pirates in the NL East. Following the season, he was dealt to the Los Angeles Dodgers, but was released during spring training. Though he never made a regular season appearance with the Dodgers, his final baseball card was #630T in the 1974 Topps Traded series, which depicted him as a Dodger.

After retirement, he operated the Outfielder’s Lounge near Shea Stadium. Agee was also known as the most active former Met, taking part in many charitable events and children’s baseball clinics around both the New York area and Mobile. He appeared as himself in a 1999 episode of Everybody Loves Raymond along with several other members of the 1969 Mets.

Agee suffered a heart attack while leaving a Midtown Manhattan office building on January 22, 2001, and died later that day at Bellevue Hospital Center, aged 58. He was buried in Pine Crest Cemetery in Mobile, Alabama. Agee was posthumously inducted into the New York Mets Hall of Fame in 2002.