Mitchell was born in Chicago, Illinois. He also grew up in the Chicago area, where he played saxophone and clarinet at around age twelve. His family was always involved in music with many different styles playing in the house when he was a child as well as having a secular music background. His brother, Norman, in particular was the one who introduced Mitchell to jazz. While attending Englewood High School in Chicago, he furthered his study of the clarinet. In the 1950s, he joined the United States Army, during which time he was stationed in Heidelberg, Germany and played in military parades with fellow saxophonists Albert Ayler and Rubin Cooper, the latter of whom, Mitchell commented, “took me under his wing and taught me a lot of stuff”. He also studied under the first clarinetist of the Heidelberg Symphony while in Germany. Mitchell returned to the United States in the early 1960s, relocated to the Chicago area, and performed in a band with Wilson Junior College undergraduates Malachi Favors (bass), Joseph Jarman, Henry Threadgill, and Anthony Braxton (all saxophonists). Mitchell also studied with Muhal Richard Abrams and played in his band, the Muhal Richard Abrams’ Experimental Band, starting in 1961.

In 1965, Mitchell was one of the first members of the non-profit organization Association for the Advancement of Creative Musicians (AACM) along with Jodie Christian (piano), Steve McCall (drums), and Phil Cohran (composer). The following year Mitchell, Lester Bowie (trumpet), Kalaparusha Maurice McIntyre (tenor saxophone), Favors, Lester Lashley (trombone), and Alvin Fielder (drums), recorded their first studio album, Sound. The album was “a departure from the more extroverted work of the New York-based free jazz players” due in part to the band recording with “unorthodox devices” such as toys and bicycle horns.

From 1967 Mitchell, Bowie, Favors and, on occasion, Jarman performed as the Roscoe Mitchell Art Ensemble, then the Art Ensemble, and finally in 1969 were billed as the Art Ensemble of Chicago. The group included Phillip Wilson on drums for short span before he joined Paul Butterfield’s band. The group lived and performed in Europe from 1969 to 1971, though they arrived without any percussionist after Wilson left. To fill the void, Mitchell commented that they “evolved into doing percussion ourselves”. The band did eventually get a percussionist, Don Moye, who Mitchell had played with before and was living in Europe at that time. For performances, the band often wore brilliant costumes and painted their faces. The Art Ensemble of Chicago have been described as becoming “possibly the most highly acclaimed jazz band” in the 1970s and 1980s.

Mitchell and the others returned to the States in 1971. After having been back in Chicago for three years, Mitchell then established the Creative Arts Collective (CAC) in 1974 that had a similar musical aesthetic to the AACM. The group was based in East Lansing, Michigan and frequently performed in auditoriums at Michigan State University. Mitchell also formed the Sound Ensemble in the early 1970s, an “outgrowth of the CAC” in his words, that consisted mainly of Mitchell, Hugh Ragin, Jaribu Shahid, Tani Tabbal, and Spencer Barefield.

In the 1990s, Mitchell started to experiment in classical music with such composers/artists such as Pauline Oliveros, Thomas Buckner, and Borah Bergman, the latter two of which formed a trio with Mitchell called Trio Space. Buckner was also part of another group with Mitchell and Gerald Oshita called Space in the late 1990s. He then conceived the Note Factory in 1992 with various old and new collaborators as another evolution of the Sound Ensemble.

He lived in the area of Madison, Wisconsin and performed with a re-assembled Art Ensemble of Chicago. In 1999, the band was hit hard with the death of Bowie, but Mitchell fought off the urge to recast his position in the group, stating simply “You can’t do that” in an interview with Allaboutjazz.com editor-in-chief Fred Jung. The band continued on despite the loss.

In 2007, Mitchell was named Darius Milhaud Chair of Composition at Mills College in Oakland, California, where he currently lives. Mitchell was chosen by Jeff Mangum of Neutral Milk Hotel to perform at the All Tomorrow’s Parties festival in March 2012 in Minehead, England.