AI company Sesame has introduced the foundational model that drives Maya, an impressively lifelike voice assistant. This model, comprising 1 billion parameters (where “parameters” refer to the distinct components of the model), is released under an Apache 2. 0 license, allowing for commercial use with minimal restrictions. Named CSM-1B, the model produces “RVQ audio codes” from both text and audio inputs, as per Sesame’s description on the AI development platform Hugging Face. RVQ stands for “residual vector quantization, ” a method used to encode audio into discrete tokens known as codes. This technique is utilized in various contemporary AI audio technologies, including Google’s SoundStream and Meta’s Encodec. CSM-1B is built on a model from Meta’s Llama family and incorporates an audio “decoder” component. According to Sesame, a fine-tuned version of CSM powers Maya. In CSM-1B’s Hugging Face and GitHub repositories, Sesame states, “The model open-sourced here is a base generation model. It can produce various voices but has not been fine-tuned for any specific voice.

[…] The model has some capability for non-English languages due to data contamination in the training set, but its performance is expected to be suboptimal. ” It remains unclear what data Sesame utilized to train CSM-1B, as the company has not disclosed this information. It’s important to note that the model lacks substantial safeguards. Sesame relies on an honor system, encouraging developers and users to refrain from using the model to imitate someone’s voice without permission, create misleading material like false news, or engage in harmful or malicious activities. I tested the demo on Hugging Face, and mimicking my voice took less than a minute. After that, generating speech on various topics, including contentious ones like elections and Russian propaganda, was straightforward. Consumer Reports recently cautioned that many popular AI-driven voice cloning tools available today lack “meaningful” protections against fraud or misuse. Sesame, co-founded by Oculus co-creator Brendan Iribe, gained significant attention in late February for its assistant technology, which nearly navigates the uncanny valley. Maya and Sesame’s other assistant, Miles, breathe, exhibit speech disfluencies, and can be interrupted while talking, similarly to OpenAI’s Voice Mode. The company has secured an undisclosed investment from Andreessen Horowitz, Spark Capital, and Matrix Partners. Beyond developing voice assistant technology, Sesame is prototyping AI glasses “intended for all-day wear” that will utilize its proprietary models.