tokyotech-llm/Llama-3.1-8B-code-ablation-exp3-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0005000
The tokyotech-llm/Llama-3.1-8B-code-ablation-exp3-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0005000 is an 8 billion parameter Llama-3.1 model continually pre-trained by tokyotech-llm. It was trained on 50 billion tokens, including 16% pylint-filtered Python code from The-Stack-v2 and 84% multilingual text, with a 32K context length. This model is specifically designed to evaluate the impact of pylint filtering on code generation performance within the SwallowCode ablation experiments, making it suitable for research in code-focused language modeling.
Loading preview...
Model Overview
This model, developed by tokyotech-llm, is a continually pre-trained Llama-3.1-8B variant. It was trained on a substantial 50 billion tokens, with a specific focus on evaluating the effects of data filtering on code generation. The training mix included 16% pylint-filtered Python code from The-Stack-v2 (SwallowCode Experiment 3) and 84% multilingual text, encompassing Japanese and English corpora. The model utilizes a Llama-3 tokenizer and was trained using Megatron-LM on NVIDIA H100 GPUs.
Key Capabilities and Purpose
- Code Generation Research: Primarily intended for research into code generation, specifically to assess the performance impact of pylint filtering on Python code datasets.
- Text Completion: Capable of text completion in both English and Japanese.
- Ablation Study: Forms part of the SwallowCode ablation models, contributing to the understanding of data preparation techniques for code-focused LLMs.
- Not Instruction-Tuned: This model is a base pre-trained model and is not instruction-tuned, making it best suited for research and further fine-tuning.
Training Details
- Architecture: Llama-3.1 with 8 billion parameters.
- Pretraining Data: 50 billion tokens, including 8 billion code tokens (pylint-filtered Python) and 42 billion multilingual text tokens (Japanese Wikipedia, Swallow Corpus v2, Laboro-ParaCorpus, English Wikipedia, Cosmopedia, DCLM).
- Context Length: Supports a sequence length of 8,192 tokens.
Evaluation
The model's performance was evaluated using the setup described in the SwallowCode paper, employing lm-evaluation-harness and BigCodeBench. Benchmarks covered include code generation tasks (HumanEval, HumanEval+) and general language understanding tasks (OpenBookQA, TriviaQA, HellaSwag, SQuAD 2.0, XWINO, MMLU, GSM8K, BBH).