Nvidia's new "Fugatto" model enhances generative AI by transforming music, voices, and sounds, even creating previously unheard sounds. Not yet publicly available, examples on the website showcase its ability to modify audio traits, from saxophones sounding like barking to underwater speech or choirs of ambulance sirens. This broad capability has led Nvidia to describe Fugatto as a "Swiss Army knife for sound. " The challenge lies in crafting a training dataset that highlights meaningful relationships between audio and language. Nvidia's researchers, using an LLM-generated Python script, created numerous template-based and free-form instructions to describe audio "personas. " These were applied to a wide array of open-source audio datasets, annotating them with natural language descriptions quantified by emotion, gender, and speech quality. The researchers held certain factors constant while varying others to teach the model distinctions like happier speech or different instrument sounds. After processing 20 million samples (50, 000 hours of audio), they used Nvidia tensor cores to develop a model with 2. 5 billion parameters, showcasing reliable audio quality scores. Beyond training, Fugatto's "ComposableART" system allows customizable audio output. It combines traits from its dataset to create new, unheard sounds, using "conditional guidance" for unseen combinations.

While not all outputs are pitch-perfect, the variety of sounds, like a violin sounding like a laughing baby, showcases Fugatto's transformative ability. Crucially, Fugatto treats audio traits as tunable continuums, not binaries. It combines sounds, like an acoustic guitar with running water, by altering the balance, and adjusts accents or emotions in speech. It performs tasks such as altering spoken text emotion, isolating vocal tracks, and replacing notes in MIDI music with varied vocal performances. Nvidia sees Fugatto as a step toward unsupervised multitask learning and envisions applications in song prototyping and dynamic video game scores. Such models are intended as tools for audio artists rather than replacements. As producer/songwriter Ido Zmishlany states, technology continuously reshapes music, with AI marking a new chapter in musical innovation.