tokyotech-llm/Llama-3.1-8B-code-ablation-exp1-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0005000
tokyotech-llm/Llama-3.1-8B-code-ablation-exp1-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0005000 is an 8 billion parameter Llama-3.1 based language model developed by tokyotech-llm. It was continually pre-trained on 50 billion tokens, including a significant portion of Python code from The-Stack-v2 and multilingual text. This model serves as a baseline for evaluating code generation performance and general capabilities, specifically designed for ablation studies within the SwallowCode project.
Loading preview...
Model Overview
This model, developed by tokyotech-llm, is a continually pre-trained version of the Llama-3.1-8B architecture. It was trained on a total of 50 billion tokens, with a specific mix of 16% Python code from The-Stack-v2 (SwallowCode, Experiment 1) and 84% multilingual text datasets. The primary purpose of this model is to establish a baseline performance for unfiltered Python code within the SwallowCode ablation experiments.
Key Capabilities and Characteristics
- Code Generation: Demonstrates baseline performance on code generation benchmarks like HumanEval and HumanEval+.
- General Language Understanding: Maintains strong general capabilities across knowledge, reasoning, and common sense benchmarks (e.g., OpenBookQA, MMLU, GSM8K).
- Training Data: Utilizes a unique blend of 8 billion Python code tokens and 42 billion multilingual text tokens, including Japanese and English corpora.
- Architecture: Based on Llama-3.1, with a sequence length of 8,192 tokens and trained using bfloat16 precision.
Use Cases
This model is particularly suitable for:
- Code Generation Tasks: As a foundational model for generating Python code.
- Research and Ablation Studies: Serving as a critical reference point for comparing the impact of different code pre-training strategies within the SwallowCode project.
- General NLP Applications: Where a balance of code understanding and general language capabilities is required.