tokyotech-llm/Llama-3.1-8B-code-ablation-exp3-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0007500
The tokyotech-llm/Llama-3.1-8B-code-ablation-exp3-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0007500 is an 8 billion parameter Llama-3.1 model continually pre-trained on 50 billion tokens, including 16% pylint-filtered Python code from The-Stack-v2 and 84% multilingual text. Developed by tokyotech-llm, this model focuses on evaluating the impact of pylint filtering on code generation performance. It is intended for research in code generation and text completion in English and Japanese, with a 32768 token context length.
Loading preview...
Model Overview
This model, developed by tokyotech-llm, is a Llama-3.1-8B variant that underwent continual pre-training on 50 billion tokens. Its primary purpose is to evaluate the performance impact of pylint-filtered Python code within the SwallowCode ablation experiments. The training dataset comprised 16% pylint-filtered Python code from The-Stack-v2 (Experiment 3) and 84% multilingual text, including significant portions of Japanese and English corpora.
Key Capabilities
- Code Generation: Optimized for code generation tasks, particularly in Python, due to its specialized training data.
- Multilingual Text Completion: Capable of text completion in both English and Japanese.
- Research Focus: Designed for research into the effects of data filtering on large language model performance.
Training Details
The model was trained using Megatron-LM on 64 NVIDIA H100 GPUs. It utilizes a Llama-3 tokenizer and supports a sequence length of 8,192 tokens. Evaluation was conducted using lm-evaluation-harness and BigCodeBench, covering benchmarks like HumanEval, HumanEval+, MMLU, and GSM8K.
Intended Use
This model is best suited for:
- Research purposes: Specifically for studying code generation and the impact of data curation techniques.
- Text completion: For generating text in English and Japanese.
- Code-related tasks: Leveraging its training on a significant volume of filtered Python code. It is not instruction-tuned.