Born in 1958, Kramer initially chose film school as his creative path, but wound up dropping out twice; instead, he worked as a sound engineer for several late-'70s New York bands and eventually joined the band that became Eugene Chadbourne's Shockabilly as a bass player and sometime organist. Upon Shockabilly's mid-'80s dissolution, Kramer served a brief stint as the Butthole Surfers' touring bassist, then decided to concentrate on running his Shimmy-Disc label and Noise New York (later Noise New Jersey) recording studio. As well as being a member of the twistedly humorous bands B.A.L.L. (later Gumball) and Bongwater, Kramer produced acts ranging from the label's own GWAR and King Missile to Galaxie 500, Urge Overkill, Half Japanese, Daniel Johnston, Fred Frith, Palace Brothers, Low, and Royal Trux, while also releasing albums by Ween, the Boredoms, John Zorn's Naked City, Damon & Naomi, and many more.

From the late '80s on, Kramer was a highly active collaborator, releasing projects with Jad Fair of Half Japanese, John S. Hall of King Missile, Ralph Carney and Daved Hild, ex-Gong frontman Daevid Allen, ex-King Missile member Dogbowl, ex-Soft Machine bassist Hugh Hopper, and Captain Howdy, a band featuring magician Penn Jillette on vocals and cartoon voice-over master Billy West (Ren & Stimpy, etc.) on guitar. Kramer's solo career began in 1993 with the release of The Guilt Trip, which was followed in 1994 with The Secret of Comedy and 1995's Japanese-only Music for Crying; the albums combined Kramer's warped humor with his ample production skills and (sometimes) pop songwriting sense. In 1998, Kramer returned to solo recording with Songs from the Pink Death on Shimmy-Disc and Let Me Explain Something to You About Art on Tzadik. A second Kramer release on Tzadik, The Greenberg Variations, arrived in 2003. ~ Steve Huey