BangorAI/Mistral-7B-Cymraeg-Welsh-v2
BangorAI/Mistral-7B-Cymraeg-Welsh-v2 is a 7 billion parameter Mistral-based chat/instruct model developed by BangorAI, continually pre-trained and fine-tuned for bilingual capabilities in both English and Welsh. It leverages the Mistral-7B-v0.1 architecture and was fine-tuned on the alpaca-cleaned dataset in both languages. This model is specifically designed for conversational tasks requiring fluency in both English and Welsh, making it ideal for applications targeting Welsh-speaking communities.
Loading preview...
Overview
BangorAI/Mistral-7B-Cymraeg-Welsh-v2 is a 7 billion parameter bilingual chat/instruct model built upon the Mistral-7B-v0.1 architecture. Developed by BangorAI, this model has undergone significant training to achieve proficiency in both English and Welsh.
Key Capabilities
- Bilingual Fluency: Trained and fine-tuned in both English and Welsh, enabling seamless conversation in either language.
- Instruction Following: Functions as an instruct model, capable of responding to user prompts and system instructions.
- Llama-2 Prompt Format: Utilizes the Llama-2 prompt format, where the system prompt's language guides the model's response language.
Training Details
The model's development involved two main stages:
- Continual Pre-training: Based on
BangorAI/mistral-7b-cy-epoch-2, which itself was a continual pre-training ofMistral-7B-v0.1using Welsh data from theallenai/MADLAD-400dataset for two epochs. - Fine-tuning: Further fine-tuned using the
yahma/alpaca-cleaneddataset, incorporating both Welsh and English data for an additional two epochs.
Use Cases
This model is particularly well-suited for applications requiring a conversational AI that can interact effectively in both English and Welsh. It's an experimental LLM, and users should exercise caution regarding factual accuracy.
An online demo is available at https://demo.bangor.ai.