Arcee-VyLinh: A Compact, High-Performance Vietnamese LLM

Arcee-VyLinh, developed by arcee-ai, is a 3 billion parameter instruction-following model based on the Qwen2.5-3B architecture, featuring a 32K token context length. It stands out for its specialized optimization for the Vietnamese language, achieving strong performance comparable to larger 4B-8B parameter models despite its compact size.

Key Capabilities & Training Innovations

• Vietnamese Language Mastery: Engineered for exceptional performance on complex Vietnamese language tasks, including chat, text generation, and question answering.
• Efficient Architecture: Leverages a 3B parameter count for efficient deployment while maintaining strong instruction-following capabilities.
• Advanced Training: Utilizes a multi-stage training process that includes:
  • Starting with Qwen2.5-3B as the base.
  • Generating 20K challenging questions using EvolKit for hard question evolution.
  • Supervised fine-tuning (SFT) to create VyLinh-SFT.
  • Proprietary model merging techniques.
  • Six epochs of iterative Direct Preference Optimization (DPO) using the ORPO-Mix-40K (Vietnamese) dataset.
• Benchmark Performance: Demonstrates competitive results on the Vietnamese subset of m-ArenaHard (CohereForAI), as judged by Claude 3.5 Sonnet.

Intended Use Cases

• Vietnamese language chat and instruction following.
• Text generation and completion in Vietnamese.
• Question answering for Vietnamese content.
• General language understanding tasks in Vietnamese.
• Content creation and summarization in Vietnamese.

Limitations

While highly capable, Arcee-VyLinh's primary focus is on Vietnamese language, and it may not perform optimally for highly specialized technical domains or exhibit occasional hallucinations on cultural-specific content.