Brooks came out of a New York music scene that was crackling with activity in the early 1960s. One of the younger players on his instrument, he was a contemporary of Felix Pappalardi and Andy Kulberg and other eclectic bass players in their late teens and early 20s, who saw a way to bridge the styles of folk, blues, rock, and jazz.
Al Kooper gave Brooks his first boost to fame when he asked him to play as part of Bob Dylan's backing band on the sessions that yielded the album Highway 61 Revisited (1965) — in contrast to the kind of folkie-electric sound generated by the band on his previous album, Bringing It All Back Home (1965). Producer Bob Johnson and Dylan were looking for a harder, in-your-face electric sound, and Brooks, along with guitarist Michael Bloomfield and organist Al Kooper, provided exactly what was needed on one of the most famous recordings of the 1960s. Brooks was also part of Dylan's early backing band which performed to great notoriety at Forest Hills, Queens and the Hollywood Bowl in 1965. This band also included Robbie Robertson (guitar), Al Kooper (keyboards) and Levon Helm (drums).
From the Dylan single and album, which became two of the most widely heard (and, at the time, most controversial) records of the 1960s, Brooks branched out in a multitude of directions, as he went on to play on records by folk artists like Eric Andersen at Vanguard Records, Richie Havens and Jim & Jean at Verve Records, transitional electric folk-rockers such as David Blue (whose producer was looking for a sound similar to that on Highway 61 Revisited), and various blues-rock fusion projects involving Bloomfield and Kooper.
It was through his participation in The Electric Flag, an extension of Michael Bloomfield and Barry Goldberg's interests in blues, that Brooks's career took an unexpected turn. The Flag only lasted in its original line-up for about a year, and much of that time was spent recording a sound track album to the film The Trip (1967) starring Peter Fonda, Dennis Hopper, Susan Strasberg, and written by Jack Nicholson and The Electric Flag, An American Music Ban'.
After the Electric Flag disbanded Brooks took up temporary residence at the Chateaux Marmont Hotel on the sunset strip. From there he recorded on Cass Elliot's Dream a Little Dream (1968) and also on The Doors The Soft Parade (1969) album on the songs "Touch Me", "Tell All the People", "Wild Child" and "Wishful Sinful". Producer Paul Rothchild wanted to give the Doors a fresh sound. He hired Harvey to play and help organize the rhythm tracks and Paul Harris to write some string and horn arrangements. Harvey also played live with the Doors at the Forum in Los Angeles and Madison Square Garden in New York and was very visible on the Michael Bloomfield/Al Kooper/Steve Stills Super Session (1968) release, one of the iconic records of late 1960s rock music. His song "Harvey's Tune" appeared on this album.
He moved east to meet Crosby, Stills & Nash in Sag Harbor, New York. Brooks and Paul Harris moved to John Sebastian's house for pre-production on John B. Sebastian (1970) album. The album was recorded at the Hit Factory.
After buying a loft in what was to become Soho, Brooks got a call from Jack Gold, vice president of Columbia Records, offering him a job as staff producer. He connected with producer Teo Macero, who led him to Miles Davis. Brooks contributed to Bitches Brew (1970) and Big Fun (1974). On the sessions in August and November 1969, two bassists were used. Brooks played electric bass while Dave Holland simultaneously played acoustic bass.
From the 1970s into the mid-1990s, Brooks worked with John Martyn, the Fabulous Rhinestones, Seals & Crofts, Fontella Bass, John Sebastian, Loudon Wainwright III, John Cale, and Paul Burlison.
He has been less active since the early 1990s, having relocated to Arizona during that decade, but has continued to perform and record. He played with Donald Fagen's musical project the New York Rock and Soul Revue from 1989–1992.
In 2006, Light in the Attic, a Seattle-based record label, reissued the 1971 album In My Own Time by Karen Dalton, which was arranged and produced by Brooks. His current band is the 17th Street Band based in Tucson, Arizona. With his wife Bonnie and guitarist Tom Kusian he started 17th Street Records'with two releases in November 2009, distributed by City Hall Records: Positively 17th Street by the 17th Street Band and El Regalo, the Gift by Francisco Gonzalez.
Brooks and his wife moved to Israel on August 4, 2009, living in Jerusalem writing his memoirs, and teaching, performing and recording in Jerusalem and Tel Aviv. Harvey & Bonnie continue their musical blog. He was featured in a cover story at Bass Musician magazine, March 2011. His wife has written a children's book, Gramps Has a Ponytail (2012), using Brooks as a model for her musical gramps.