Synatra-7B-v0.3-dpo Overview
Synatra-7B-v0.3-dpo is a 7 billion parameter language model developed by maywell, built upon the mistralai/Mistral-7B-Instruct-v0.1 base model. It has been trained on a single A100 80GB GPU and is designed to follow both ChatML and Alpaca (No-Input) instruction formats, ensuring broad compatibility with existing conversational AI pipelines.
Key Capabilities & Performance
This model demonstrates notable performance in Korean language understanding benchmarks. On the KOBEST_BOOLQ, SENTINEG, and WIC zero-shot evaluations using EleutherAI/lm-evaluation-harness, Synatra-7B-v0.3-dpo achieved a BoolQ score of 0.8058 and a SentiNeg score of 0.8942, outperforming EleutherAI/polyglot-ko-12.8b in BoolQ. While its COPA and HellaSwag scores are slightly lower than some alternatives, its strength in specific Korean understanding tasks is a key differentiator.
Further evaluation on the Open LLM Leaderboard shows an average score of 53.14, with specific metrics including:
- ARC (25-shot): 62.8
- HellaSwag (10-shot): 82.58
- MMLU (5-shot): 61.46
- TruthfulQA (0-shot): 56.46
- Winogrande (5-shot): 76.24
Good For
- Korean Language Applications: Excels in tasks requiring understanding and generation of Korean text, particularly for question answering (BoolQ) and sentiment analysis (SentiNeg).
- Conversational AI: Its adherence to ChatML and Alpaca instruction formats makes it suitable for chatbots and dialogue systems.
- Developers using Mistral-7B-Instruct-v0.1: Provides a fine-tuned alternative with enhanced Korean language capabilities.