bertin-project/Gromenauer-7B
Gromenauer-7B by bertin-project is a 7 billion parameter Spanish language model built on the Mistral architecture. It is specifically designed to understand and generate high-quality Spanish text, having been trained on an extensive literary corpus. This model excels at capturing linguistic nuances, styles, and contexts found in Spanish literature, making it ideal for Spanish-centric natural language processing tasks.
Loading preview...
Overview
bertin-project's Gromenauer-7B is a 7 billion parameter language model based on the Mistral architecture, specifically engineered for the Spanish language. It has been trained on an extensive literary corpus to ensure high-quality text generation and a deep understanding of Spanish linguistic nuances, styles, and contexts.
Key Capabilities
- High-Quality Spanish Text Generation: Optimized for producing fluent and contextually appropriate Spanish text.
- Linguistic Nuance Capture: Training on a broad literary corpus allows it to grasp subtle linguistic variations.
- Mistral Architecture Benefits: Leverages the efficiency and performance characteristics of the Mistral model family, including features like Flash Attention and Sliding Window Attention.
Technical Specifications
- Model Type: Mistral
- Sequence Length: 8192 tokens
- Hidden Dimension: 4096
- Number of Layers: 32
- Attention Heads: 32 (8 Key-Value Heads)
- Sliding Window Attention: 4096 tokens
Training Details
- Tokenizer: Uses the
mistralai/Mistral-7B-v0.1tokenizer. - Learning Rate: 1e-5 with Adam optimizer and Cosine schedule.
- Training Steps: 7000 steps.
Good For
- Applications requiring robust Spanish language understanding and generation.
- Tasks involving literary analysis or style transfer in Spanish.
- Developers seeking a specialized, high-performance Spanish LLM.