Model Overview
neph1/bellman-7b-mistral-instruct-v0.2 is a 7 billion parameter language model built upon the Mistral architecture, developed by neph1. This version has undergone Qlora training for about two epochs on a diverse dataset of 14,000 rows, including question-and-answer pairs, Python code examples, and general instruction-following tasks. It features a 4096 token context length and has received a light DPO (Direct Preference Optimization) pass to enhance alignment and reduce factual inaccuracies.
Key Capabilities
- Swedish Language Proficiency: Specifically trained to improve knowledge and quality in Swedish topics and language.
- Instruction Following: Fine-tuned on a broad range of instruction-type questions for general utility.
- Code Understanding: Includes Python examples in its training data, suggesting some capability in code-related tasks.
- Alignment: Utilizes a DPO pass to make the model less prone to generating untrue statements.
Training Details
- Base Model: Mistral-7B-Instruct-v0.2.
- Training Method: Qlora for approximately 2 epochs.
- Dataset Size: 14,002 rows, with increased diversity in the instruct dataset.
- Context Length: 4096 tokens.
- Configuration: Rank 64, Alpha 128, Learning rate 3e-5.
Good For
- Applications requiring strong performance in Swedish language generation and understanding.
- Instruction-following tasks where factual accuracy and reduced hallucination are important.
- Use cases benefiting from a 7B parameter model with a 4K context window for efficiency and performance.