Frazier was born in Spiro, Oklahoma but was raised in Bakersfield, California. As a teenager, he played with Ferlin Husky and on the program Hometown Jamboree; and released his first single, “Space Command”, at age 14 in 1954. As he told writer Edd Hurt in a 2008 profile for the music website Perfect Sound Forever, “We were part of The Grapes of Wrath. We were the Okies who went out to California with mattresses tied on the tops of their Model A Fords. My folks were poor. At twelve I moved away from home, with my folks’ permission. Ferlin [Husky] offered me a job, and I started working with him when I was twelve. Then I recorded a side for Capitol Records when I was fourteen, and I did some country. I cut in the big circular building that’s still out there on Hollywood and Vine.”

Frazier’s 1957 song “Alley Oop”, later taken to No. 1 in the US by The Hollywood Argyles, was his first hit. After Hometown Jamboree went off the air, Frazier moved to Nashville, Tennessee and found work as a songwriter. Among his early successes was “Timber I’m Falling”, a hit for Husky in 1964, and “There Goes My Everything”, a big hit for Jack Greene in 1966 that earned him a nomination for the Grammy Award for Best Country Song.

In 1966 he released his solo debut album Elvira, containing his song “Elvira”. His follow-up, Tell It Like It Is (1967), was also a success.

While his singing success was limited, Frazier became an oft-covered songwriter. His tunes were recorded by O.C. Smith, George Jones (who recorded an entire album of Frazier’s songs in 1968), Diana Ross, Engelbert Humperdinck, Jerry Lee Lewis, Jack Greene, Connie Smith (who also recorded an entire album of Frazier’s songs in 1972), Willie Nelson, Brenda Lee, Carola, Charley Pride, Merle Haggard, Gene Watson, Elvis Presley, Moe Bandy, Roy Head, Charlie Louvin, Rodney Crowell, Dan McCafferty, Poco, and Ronnie Hawkins. In 1970, Frazier earned his second Grammy nomination for Best Country Song, which is awarded to the songwriter rather than the performer, for “All I Have to Offer You (Is Me), which became a #1 hit for Charley Pride. Many of the songs became hits into the 1980s; examples include the Oak Ridge Boys cover of “Elvira” and Emmylou Harris’s version of “Beneath Still Waters”. The cover of “Elvira” by the Oak Ridge Boys was a huge crossover hit, peaking at No. 1 on Billboard’s Hot Country Singles and Tracks chart and No. 5 on the all genre Billboard Hot 100 and earned Frazier his third Grammy nomination for Best Country song. Anne Murray with Glen Campbell, George Strait, Randy Travis, and Patty Loveless have all also recorded Frazier tunes. Frazier himself charted eight times on the U.S. country charts.

He was inducted into the Nashville Songwriters Hall of Fame in 1976.

In 1988, Frazier left the music industry and became a minister.