ryota-komatsu/SylReg-LM-7B-Instruct
SylReg-LM-7B-Instruct by ryota-komatsu is a 7.6 billion parameter Qwen2ForCausalLM instruction-tuned model with a 32768 token context length. It is specifically designed for speech language modeling, capable of processing syllabic units derived from audio inputs and generating corresponding speech units. This model excels in unit-to-speech synthesis, enabling the generation of speech from encoded syllabic representations.
Loading preview...
SylReg-LM 7B Instruct: Speech Language Model
SylReg-LM-7B-Instruct, developed by ryota-komatsu, is a 7.6 billion parameter Qwen2ForCausalLM instruction-tuned model. It is built upon the ryota-komatsu/SylReg-LM-7B base model and is designed for advanced speech language modeling tasks, operating with a substantial 32768 token context length.
Key Capabilities
- Syllabic Unit Processing: The model can encode raw waveforms into discrete syllabic units using a
SylRegForSyllableDiscoveryencoder. - Speech Language Modeling: It processes sequences of these syllabic units to generate new sequences, effectively performing language modeling directly on speech components.
- Unit-to-Speech Synthesis: Coupled with a
FlowMatchingWithBigVGandecoder, the model can convert generated syllabic units back into audible speech waveforms. - English Language Support: Primarily focused on English, leveraging a diverse training dataset.
Training Details
The model was trained on a massive dataset totaling over 129,364 hours of audio, including prominent datasets like LibriSpeech, Libriheavy, Emilia-Large, People's Speech, and VoxPopuli, alongside text datasets such as TinyStories and Cosmopedia-v2. Training involved 15k steps using bf16 mixed precision on 32 NVIDIA H100 GPUs.
Good for
- Researchers and developers working on novel speech synthesis methods.
- Applications requiring direct manipulation and generation of speech at a syllabic level.
- Experiments in speech language modeling and understanding the relationship between syllabic units and generated speech.