tokyotech-llm/Llama-3.1-8B-code-ablation-exp3-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0012500
The tokyotech-llm/Llama-3.1-8B-code-ablation-exp3-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0012500 is an 8 billion parameter Llama-3.1 model continually pre-trained on 50 billion tokens, including 16% pylint-filtered Python code from The-Stack-v2 and 84% multilingual text. Developed by tokyotech-llm, this model is designed for research into the effects of code filtering on performance, focusing on code generation and text completion in English and Japanese. It is not instruction-tuned and serves as an ablation experiment within the SwallowCode project, evaluating specific training data compositions.
Loading preview...
Model Overview
This model, tokyotech-llm/Llama-3.1-8B-code-ablation-exp3-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0012500, is an 8 billion parameter Llama-3.1 architecture that underwent continual pre-training. It was trained on a total of 50 billion tokens, with a specific focus on evaluating the impact of pylint-filtered Python code within the SwallowCode ablation experiments.
Key Characteristics
- Architecture: Based on Llama-3.1, with 8 billion parameters.
- Training Data: Continually pre-trained on a mix of 16% pylint-filtered Python code from The-Stack-v2 (Experiment 3 of SwallowCode) and 84% multilingual text (Japanese Wikipedia, Swallow Corpus v2, Laboro-ParaCorpus, English Wikipedia, Cosmopedia, DCLM).
- Context Length: Supports a sequence length of 8,192 tokens.
- Research Focus: Primarily intended for research purposes to understand the performance implications of pylint filtering in code datasets, as detailed in the SwallowCode paper.
- Not Instruction-Tuned: This model is not instruction-tuned, making it suitable for text completion tasks rather than direct instruction following.
Intended Use Cases
- Code Generation: Excels in code generation tasks, particularly for Python, due to its specialized training on pylint-filtered code.
- Text Completion: Capable of text completion in both English and Japanese.
- Research & Evaluation: Ideal for researchers studying the effects of data composition and filtering on large language model performance, especially within the context of code-focused pre-training. It is part of the SwallowCode ablation models.