tokyotech-llm/Llama-3.1-Swallow-8B-v0.1

Llama 3.1 Swallow 8B v0.1: Enhanced Japanese and English Capabilities

Llama 3.1 Swallow 8B v0.1 is an 8 billion parameter large language model developed by tokyotech-llm, based on the Meta Llama 3.1 architecture. This model underwent extensive continual pre-training using approximately 200 billion tokens from diverse sources, including the Swallow Corpus Version 2 (a large Japanese web corpus), Japanese and English Wikipedia, and specialized mathematical and coding datasets. The primary goal of this development was to significantly enhance the model's Japanese language proficiency while preserving its strong English language capabilities.

Key Capabilities

• Bilingual Proficiency: Excels in both Japanese and English language understanding and generation.
• Continual Pre-Training: Leverages a broad dataset for improved performance across various domains.
• Llama 3.1 Foundation: Benefits from the robust architecture and general capabilities of the Meta Llama 3.1 base model.
• Context Length: Supports a context window of 32,768 tokens.

Performance Highlights

On Japanese benchmarks, Llama 3.1 Swallow 8B v0.1 achieves a Ja Avg of 0.4905, outperforming Llama 3.1 8B (0.4359) and Llama 3 8B (0.4292). Notably, it shows strong results in JEMHopQA (0.5092), NIILC (0.6011), and WMT20-en-ja (0.2909). For English tasks, it maintains competitive performance with an En Avg of 0.5577, demonstrating its ability to retain the original Llama 3.1's English strengths while specializing in Japanese.

Good for

• Applications requiring high-quality text generation and understanding in both Japanese and English.
• Developers looking for a Llama 3.1-based model with superior Japanese language support.
• Tasks involving Japanese web content analysis, translation, or content creation.