tokyotech-llm/Llama-3.1-8B-code-ablation-exp2-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0002500
The tokyotech-llm/Llama-3.1-8B-code-ablation-exp2-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0002500 is an 8 billion parameter Llama-3.1 model continually pre-trained on 50 billion tokens, including 16% syntax-error-free Python code from The-Stack-v2 and 84% multilingual text. Developed by tokyotech-llm, this model is designed for text completion in English and Japanese, with a specific focus on code generation tasks. It serves as an ablation study model to evaluate the impact of syntax error filtering in code datasets, making it primarily suitable for research purposes.
Loading preview...
Model Overview
This model, developed by tokyotech-llm, is an 8 billion parameter Llama-3.1 architecture that underwent continual pre-training. It was trained on a total of 50 billion tokens, utilizing a unique data mix to evaluate the impact of syntax-filtered code.
Key Characteristics
- Architecture: Based on Llama-3.1, with 8 billion parameters.
- Training Data: Continually pre-trained on 50 billion tokens, comprising 16% syntax-error-free Python code from The-Stack-v2 (Experiment 2 of SwallowCode ablation) and 84% multilingual text (Japanese Wikipedia, Swallow Corpus v2, Laboro-ParaCorpus, English Wikipedia, Cosmopedia, DCLM).
- Context Length: Supports a sequence length of 8,192 tokens.
- Training Objective: Part of the SwallowCode ablation experiments, specifically designed to assess the performance implications of syntax error filtering in code datasets.
Intended Use Cases
- Code Generation: Excels in text completion for code, particularly Python, due to its specialized training on syntax-error-free code.
- Multilingual Text Completion: Capable of text completion in both English and Japanese.
- Research: Primarily intended for research purposes, especially for studying the effects of data curation strategies on code generation capabilities. It is not instruction-tuned.
Training Details
The model was trained using Megatron-LM on 64 NVIDIA H100 GPUs on the TSUBAME supercomputer. Evaluation was conducted using lm-evaluation-harness and BigCodeBench, covering benchmarks like HumanEval, HumanEval+, OpenBookQA, MMLU, and GSM8K.