tokyotech-llm/Llama-3.1-8B-code-ablation-exp1-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0012500
The tokyotech-llm/Llama-3.1-8B-code-ablation-exp1-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0012500 is an 8 billion parameter Llama-3.1 model continually pre-trained by tokyotech-llm. It was trained on 50 billion tokens, including a 16% Python code subset from The-Stack-v2 and 84% multilingual text, with a sequence length of 8,192 tokens. This model serves as a baseline for evaluating unfiltered Python code performance in code generation tasks and maintaining general language capabilities.
Loading preview...
Model Overview
This model, developed by tokyotech-llm, is a continually pre-trained version of Llama-3.1-8B. It was specifically designed as a baseline for ablation experiments within the SwallowCode project, focusing on the performance of unfiltered Python code from The-Stack-v2. The model demonstrates foundational capabilities in code generation while preserving general knowledge and reasoning abilities.
Training Details
The model underwent continual pre-training on 50 billion tokens. The training data composition was 16% Python code from The-Stack-v2-train-smol-ids (8 billion tokens) and 84% multilingual text, including Japanese and English corpora. Training utilized Megatron-LM on 64 NVIDIA H100 GPUs with a sequence length of 8,192 tokens.
Key Capabilities
- Code Generation: Provides baseline performance on benchmarks like HumanEval and HumanEval+.
- General Language Understanding: Maintains capabilities across knowledge, reasoning, and common sense tasks.
- Multilingual Support: Incorporates a significant portion of multilingual text in its training data.
Use Cases
This model is primarily intended as a reference point for researchers and developers exploring the impact of code data quality and filtering in large language models. It can be used for:
- Benchmarking code generation performance against subsequent experimental models.
- Understanding the baseline capabilities of Llama-3.1 when continually pre-trained on a specific mix of Python code and multilingual text.