Skip to the content.

This website accompanies the NeurIPS 2023 Machine Learning for Audio Workshop paper entitled “InstrumentGen: Generating Sample-Based Musical Instruments From Text.” Here, we provide listening examples of the instruments generated by the model described in the paper.

Abstract

We introduce the text-to-instrument task, which aims at generating sample-based musical instruments based on textual prompts. Accordingly, we propose InstrumentGen, a model that extends a text-prompted generative audio framework to condition on instrument family, source type, pitch (across an 88-key spectrum), velocity, and a joint text/audio embedding. Furthermore, we present a differentiable loss function to evaluate the intra-instrument timbral consistency of sample-based instruments. Our results establish a foundational baseline, paving the way for further research in the domain of automatic sample-based instrument generation.

Text-to-Instrument

Prompt Generated audio (2 octaves)
dark concert grand piano
bright upright piano
deep and punchy sub bass
distorted synth bass
distorted electric guitar lead
bright acoustic guitar
aggressive synth lead
hammond organ
warm cello
silky violin

Advanced Descriptive Prompts/Limitations

To showcase the capabilities and current limitations of the model in response to advanced descriptive prompts, we present the following examples.

Prompt Generated audio (2 octaves)
A string ensemble characterized by high harmonics, light bowing, and sparse vibrato, yielding an airy, floating tonal quality.
A bass synth with a distorted sawtooth waveform and high resonance, delivering a gritty, aggressive sonic texture.
Staccato piano notes augmented with a synthetic overlay and digital delay, producing a crisp, rhythmically precise tonal effect.

Sample-to-Instrument

Although it is not the central focus of our paper, our system inherently accommodates a sample-to-instrument functionality, whereby a musical instrument can be generated from a single audio reference as input. We notionally demonstrate this with an out-of-domain audio sample used as the prompt.

Prompt Generated audio (2 octaves)