tokyotech-llm/Llama-3.1-8B-code-ablation-exp1-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0010000
The tokyotech-llm/Llama-3.1-8B-code-ablation-exp1-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0010000 is an 8 billion parameter Llama-3.1 based model continually pre-trained on 50 billion tokens, including a 16% Python code subset from The-Stack-v2 and 84% multilingual text. Developed by tokyotech-llm, this model serves as a baseline for evaluating unfiltered Python code performance in the SwallowCode ablation experiments. It demonstrates baseline code generation capabilities while maintaining general knowledge and reasoning abilities, making it a reference point for further code-centric model comparisons.
Loading preview...
Model Overview
This model, developed by tokyotech-llm, is an 8 billion parameter Llama-3.1 based language model that underwent continual pre-training. It was trained on a total of 50 billion tokens, with a specific data mix: 16% consisted of the Python subset from The-Stack-v2-train-smol-ids (part of SwallowCode, Experiment 1), and the remaining 84% comprised various multilingual text datasets including Japanese Wikipedia, Swallow Corpus v2, Laboro-ParaCorpus, English Wikipedia, Cosmopedia, and DCLM.
Key Characteristics
- Architecture: Llama-3.1 with 8 billion parameters.
- Training Data: 50 billion tokens, with a significant portion dedicated to Python code (8 billion tokens).
- Purpose: Serves as a baseline model within the SwallowCode ablation experiments to evaluate the performance of unfiltered Python code.
- Performance: Exhibits baseline performance on code generation benchmarks like HumanEval and HumanEval+, while preserving general capabilities in knowledge, reasoning, and common sense tasks.
- Context Length: Supports a sequence length of 8,192 tokens.
Use Cases
This model is primarily intended as a reference point for researchers and developers comparing the impact of different code data filtering and training strategies within the SwallowCode pipeline. It can be used for:
- Establishing a baseline for code generation performance.
- Evaluating the effectiveness of subsequent code-centric model improvements.
- General text generation and understanding tasks where Llama-3.1 capabilities are desired.