sesame/csm-1b
CSM-1B (Conversational Speech Model) by Sesame is a 1 billion parameter speech generation model that creates RVQ audio codes from text and audio inputs. Built on a Llama backbone, it features a smaller audio decoder producing Mimi audio codes. This model excels at generating conversational audio, especially when provided with context, and supports batched inference and full-graph compilation for performance.
Loading preview...
CSM-1B: Conversational Speech Model
CSM-1B, developed by Sesame, is a 1 billion parameter speech generation model designed to produce RVQ audio codes from both text and audio inputs. It leverages a Llama-based architecture combined with a compact audio decoder that generates Mimi audio codes. This model is particularly adept at generating natural-sounding conversational audio, with performance significantly enhanced when provided with contextual audio.
Key Capabilities
- Speech Generation: Converts text and optional audio context into high-quality RVQ audio codes.
- Contextual Generation: Produces more natural and coherent speech when given prior audio conversation as context.
- Performance Optimization: Supports batched inference and full-graph compilation with CUDA graphs for accelerated generation.
- Fine-tuning: Can be fine-tuned using the Hugging Face Transformers Trainer for custom applications.
Use Cases and Limitations
CSM-1B is ideal for applications requiring conversational audio generation, such as interactive voice demos or synthetic speech for dialogues. It is a base generation model capable of producing a variety of voices, though it has not been fine-tuned for specific voices. It is important to note that CSM-1B is an audio generation model, not a general-purpose multimodal LLM; it cannot generate text. While it may exhibit some capacity for non-English languages due to training data contamination, its primary strength and intended use are for English speech generation.