Coe was born in Akron, Ohio, on September 6, 1939. His favorite singer as a child was Johnny Ace. After being sent to the Starr Commonwealth For Boys reform school at the age of 9, he spent much of the next 20 years in correctional facilities, including three years at the Ohio Penitentiary. Coe received encouragement to begin writing songs from Screamin' Jay Hawkins, with whom he had spent time in prison.
After concluding another prison term in 1967, Coe embarked on a music career in Nashville, living in a hearse which he parked in front of the Ryman Auditorium. He caught the attention of the independent record label Plantation Records and signed a contract with the label.
Early in 1970, Coe released his debut album, Penitentiary Blues, followed by a tour with Grand Funk Railroad. In October 1971 he signed as an exclusive writer with Pete and Rose Drake's publishing company Windows Publishing Company, Inc. in Nashville, Tennessee, where he remained until 1977. Although he developed a cult following with his performances, he was not able to develop any mainstream success, but other performers achieved charting success by recording songs Coe had written, including Billie Jo Spears' 1972 recording "Souvenirs & California Mem'rys" and Tanya Tucker's 1973 single "Would You Lay With Me (In a Field of Stone)," which was a number one hit, and responsible for Coe becoming one of Nashville's hottest songwriters and Coe himself being signed by Columbia Records. Coe recorded his own version of the song for his second Columbia album, Once Upon a Rhyme, released in 1975. AllMusic writer Thom Jurek said of the song, "The amazing thing is that both versions are definitive." Johnny Cash covered the song in his album American III: Solitary Man in 2000.The album also contained a cover of Steve Goodman's and John Prine's "You Never Even Called Me by My Name," which was a Top Ten Billboard hit, and was followed by a string of moderately successful hits.
Coe was a featured performer in Heartworn Highways, a 1975 documentary film by James Szalapski. Other performers featured in this film included Guy Clark, Townes Van Zandt, Rodney Crowell, Steve Young, Steve Earle, and The Charlie Daniels Band. In 1977 Johnny Paycheck released a cover of Coe's "Take This Job And Shove It," which was a number one hit and Coe's most successful song.
While Coe lived in Key West, Shel Silverstein played his comedy album Freakin' at the Freakers Ball for Coe, spurring him to perform his own comedic songs for Silverstein, who encouraged Coe to record them, leading to the production of the independently released Nothing Sacred. Jimmy Buffett accused Coe of plagiarizing the melody of "Divers Do It Deeper" from Buffett's "Changes in Latitudes, Changes in Attitudes", stating, "I would have sued him, but I didn't want to give Coe the pleasure of having his name in the paper." In response to the success of Buffett's song, Coe wrote a song insulting Buffett, and it appeared on Nothing Sacred. The album was released by mail order in 1978, through the back pages of the biker magazine Easyriders. Coe's 1979 Columbia album Spectrum VII contained a note stating "Jimmy Buffett doesn't live in Key West anymore," a lyric from a song from Nothing Sacred.
In 1982, Coe released another independent album, Underground Album, which contained his most controversial song, "Nigger Fucker," which resulted in Coe being accused of racism. Coe responded to the accusations by stating "Anyone that hears this album and says I'm a racist is full of shit." Coe's drummer, Kerry Brown, is black and married to a white woman, as was Brown's late father, Clarence "Gatemouth" Brown.
During the 1980s, Coe enjoyed a resurgence in mainstream popularity, twice hitting the top 10 of the Billboard Hot Country Singles chart with The Ride (1983) and Mona Lisa Lost Her Smile (1984). The Ride recounts a drifter's encounter with the ghost of country music legend Hank Williams. Mona Lisa is a mid-tempo ballad about a broken love affair, featuring allusions to the iconic Da Vinci painting. He also just missed the top 10 in early 1985 with She Used to Love Me a Lot.
In 1990, Coe reissued his independent albums Nothing Sacred and Underground Album on compact disc, as well as the compilation 18 X-Rated Hits. Throughout the 1990s, Coe had a successful career as a concert performer in the United States and Europe. In 1999, Coe met Pantera guitarist Dimebag Darrell in Fort Worth, Texas, and the two musicians, struck by the similarity of the approaches between country and heavy metal, agreed to work together, and began production on an album.
In 2000, Coe toured as the opening act for Kid Rock, and The New York Times published an article by journalist Neil Strauss, who described the material on Nothing Sacred and Underground Album as "among the most racist, misogynist, homophobic and obscene songs recorded by a popular songwriter." Coe maintains that he wrote to Strauss during the writing of the article, but the journalist did not acknowledge any interaction between the two, only stating that Coe's manager refused to speak on the record.
In 2003, Coe wrote a song for Kid Rock, Single Father, which appeared on Kid Rock's self-titled album, and was released as a single, which peaked at No. 50 on the Billboard Country Singles chart.
Rebel Meets Rebel, with Dimebag Darrell, Vinnie Paul and Rex Brown, recorded sporadically between 1999 and 2003, was released in 2006, two years after Darrell's murder. AllMusic described it as a "groundbreaking" country metal album.
In the 2006 video "God's Gonna Cut You Down", Coe introduces Johnny Cash as the Man in Black. The video, directed by Tony Kaye, was released in connection with Cash's cover of the song in American V: A Hundred Highways.
In 2017 he was featured singing "Take This Job" on the album, Baptized in Bourbon, by the Moonshine Bandits. He also sings in the video.
Coe's musical style derives from blues, rock and country music traditions. His vocal style is described as a "throaty baritone." His lyrical content is often humorous or comedic, with William Ruhlmann describing him as a "near-parody of a country singer." Stephen Thomas Erlewine describes Coe as "a great, unashamed country singer, singing the purest honky-tonk and hardest country of his era […] He may not be the most original outlaw, but there's none more outlaw than him."
Coe's lyrics frequently include references to alcohol and drug use, and are often boisterous and cocky. Coe's debut album, Penitentiary Blues was described as "voodoo blues" and "redneck music" by Allmusic's Thom Jurek. It focused on themes such as working for the first time, blood tests from veins used to inject heroin, prison time, hoodoo imagery and death. The album's influences included Charlie Rich, Jerry Lee Lewis, Bo Diddley, Lightnin' Hopkins, and Tony Joe White.
Coe's first country album, The Mysterious Rhinestone Cowboy, has been described as alt-country, "pre-punk" and "a hillbilly version of Marc Bolan's glitz and glitter." Credited influences on the album include Merle Haggard.
Coe's albums Nothing Sacred and Underground Album contained profane, sexually explicit material, including songs making reference to an orgy in Nashville's Centennial Park, sex with pornographic film star Linda Lovelace and insults directed at Jimmy Buffett and Anita Bryant. The album Rebel Meets Rebel featured a song, "Cherokee Cry," which criticizes the United States government's treatment of Native Americans.
In his early career, Coe was known for his unpredictable live performances, in which he would ride a Harley-Davidson motorcycle onto the stage and curse at his audience. Coe has also performed in a rhinestone suit and a mask which resembled that of The Lone Ranger, calling himself the "Mysterious Rhinestone Cowboy."