tokyotech-llm/Llama-3.1-8B-code-ablation-exp1-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0007500
This tokyotech-llm Llama-3.1-8B model is a continually pre-trained Llama-3.1-8B variant, specifically fine-tuned on a mix of 16% Python code from The-Stack-v2 and 84% multilingual text, totaling 50 billion tokens. It serves as a baseline for ablation experiments in code generation, maintaining general capabilities while providing a reference for code-specific performance. The model has a context length of 32768 tokens and is designed to evaluate the impact of unfiltered Python code on LLM performance.
Loading preview...
Model Overview
This model, developed by tokyotech-llm, is a continually pre-trained version of Meta's Llama-3.1-8B, specifically adapted for code-related research. It was trained on 50 billion tokens, comprising a unique mix of 16% Python code from The-Stack-v2 (Experiment 1 of SwallowCode) and 84% multilingual text datasets, including Japanese and English corpora. This specific configuration aims to establish a baseline performance for unfiltered Python code within the SwallowCode ablation experiments.
Key Capabilities
- Code Generation: Provides baseline performance on benchmarks like HumanEval and HumanEval+.
- General Language Understanding: Maintains capabilities across knowledge, reasoning, and common sense tasks (e.g., OpenBookQA, MMLU, GSM8K).
- Research Reference: Serves as a crucial reference point for comparing subsequent ablation studies in the SwallowCode pipeline, evaluating the impact of different code training data.
Training Details
The model was trained using Megatron-LM on 64 NVIDIA H100 GPUs, with a sequence length of 8,192 tokens and bfloat16 precision. The data mix included significant portions of Japanese and English text alongside the Python code. Evaluation was conducted using lm-evaluation-harness and BigCodeBench, with results reported at various checkpoints up to 50 billion tokens.