Synatra-V0.1-7B-Instruct Overview

Synatra-V0.1-7B-Instruct is a 7 billion parameter instruction-tuned model developed by StableFluffy, built upon the mistralai/Mistral-7B-Instruct-v0.1 base. This version is specifically licensed for non-commercial use only (cc-by-nc-4.0), which takes precedence over the Llama 2 Community License Agreement.

Key Capabilities & Performance

• Strong Korean Language Performance: The model shows competitive results in Korean language benchmarks.
  • KULLM Evaluation: Achieves high scores in understanding, naturalness, context retention, and overall quality, outperforming several other Korean models including KoAlpaca and koVicuna, and closely matching GPT-4 in some metrics.
  • KOBEST_BOOLQ: Scores 0.849, indicating strong performance in boolean question answering for Korean.
  • KOBEST_SENTINEG: Achieves 0.8690 in sentiment negation tasks.
• Instruction Following: Designed to follow instructions, though the initial version uses [\INST] instead of [/INST] due to a training error, which is slated for correction in v0.2.

Usage Notes

• Instruction Format: Prompts should be enclosed with [INST] and [\INST] tokens, with a space recommended at the end of the prompt.
• Non-Commercial License: Strictly for non-commercial applications.

Open LLM Leaderboard Evaluation

• Average Score: 53.54
• HellaSwag (10-shot): 76.63
• MMLU (5-shot): 55.29

This model is a solid choice for developers focusing on Korean language tasks within a non-commercial framework, offering a balance of performance and accessibility.