Nero10578/Mistral-7B-Sunda-v1.0
Nero10578/Mistral-7B-Sunda-v1.0 is a 7 billion parameter Mistral-7B-v0.1 fine-tune, specifically adapted for the Sundanese language. This model was developed to explore the feasibility of teaching a new language to an existing LLM with limited data and QLoRA fine-tuning. It enhances the base Mistral model's capabilities by adding Sundanese language support, making it suitable for applications requiring interaction in Sundanese. The model maintains a 4096 token context length.
Loading preview...
Model Overview
Nero10578/Mistral-7B-Sunda-v1.0 is a 7 billion parameter language model based on Mistral-7B-v0.1, fine-tuned to incorporate Sundanese language capabilities. This project aimed to demonstrate that a model can acquire a new language not inherently supported, even with limited datasets and QLoRA fine-tuning.
Key Capabilities
- Sundanese Language Support: The primary feature is the addition of Sundanese language understanding and generation, enabling the model to respond in Sundanese.
- Qlora Fine-tuning: Utilizes QLoRA for efficient adaptation, demonstrating effective language transfer with constrained resources.
- Base Model Strength: Leverages the robust architecture of Mistral-7B-v0.1, providing a strong foundation for its extended linguistic abilities.
Training Details
The model was fine-tuned using a cleaned and deduplicated Sundanese corpus derived from the w11wo/nlp-datasets repository. Training involved 2 epochs with specific hyperparameters, including a sequence length of 1024, lora_r of 8, and learning_rate of 0.0002, optimized through trial and error.
Use Cases
This model is particularly well-suited for applications requiring:
- Sundanese Chatbots: Engaging in conversations and providing information in Sundanese.
- Language Translation (Sundanese): Assisting with basic translation tasks involving Sundanese.
- Educational Tools: Supporting learning or content generation in the Sundanese language.