tokyotech-llm/Llama-3.1-8B-code-ablation-exp3-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0002500
The tokyotech-llm/Llama-3.1-8B-code-ablation-exp3-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0002500 is an 8 billion parameter Llama-3.1 based language model continually pre-trained by tokyotech-llm. It was trained on 50 billion tokens, including 16% pylint-filtered Python code from The-Stack-v2 and 84% multilingual text, with a context length of 32768 tokens. This model is specifically designed for evaluating the impact of pylint filtering on code generation performance within the SwallowCode ablation experiments. It is intended for text completion in English and Japanese, with a strong focus on code generation tasks, and is best suited for research purposes.
Loading preview...
Model Overview
This model, developed by tokyotech-llm, is a continual pre-training of the Llama-3.1-8B architecture. It was trained on 50 billion tokens, utilizing a mix of 16% pylint-filtered Python code from The-Stack-v2 (Experiment 3 of SwallowCode) and 84% multilingual text. The primary goal of this specific iteration is to evaluate the performance impact of pylint filtering on Python code within the SwallowCode ablation experiments.
Key Characteristics
- Architecture: Llama-3.1-8B with 8 billion parameters.
- Training Data: 50 billion tokens, comprising 16% pylint-filtered Python code and 84% multilingual text (Japanese and English Wikipedia, Swallow Corpus v2, Cosmopedia, DCLM, Laboro-ParaCorpus).
- Context Length: Supports a sequence length of 8,192 tokens.
- Training Details: Trained using Megatron-LM on NVIDIA H100 GPUs, with checkpoints evaluated at various token counts up to 50B.
- Evaluation: Assessed using lm-evaluation-harness and BigCodeBench, covering code generation (HumanEval, HumanEval+) and general tasks (MMLU, GSM8K, etc.).
Intended Use Cases
- Code Generation: Excels at code generation tasks, particularly in Python, due to its specialized training on pylint-filtered code.
- Text Completion: Capable of text completion in both English and Japanese.
- Research: Primarily intended for research purposes, especially for understanding the effects of data filtering strategies in code LLM pre-training. It is not instruction-tuned.