tokyotech-llm/Llama-3.1-8B-code-ablation-exp2-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0005000
The tokyotech-llm/Llama-3.1-8B-code-ablation-exp2-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0005000 is an 8 billion parameter Llama-3.1 model continually pre-trained by tokyotech-llm. It was trained on 50 billion tokens, including 16% syntax-filtered Python code from The-Stack-v2 and 84% multilingual text, to evaluate the impact of syntax error filtering on code generation. This model is intended for research into code generation and text completion in English and Japanese, offering a 32768 token context length.
Loading preview...
Model Overview
This model, developed by tokyotech-llm, is a Llama-3.1-8B architecture that underwent continual pre-training. It was specifically designed for the SwallowCode ablation experiments to assess the performance impact of using syntax-filtered Python code from The-Stack-v2.
Key Characteristics
- Architecture: Llama-3.1 with 8 billion parameters.
- Training Data: Continually pre-trained on 50 billion tokens, comprising 16% syntax-error-free Python code (from SwallowCode Experiment 2) and 84% multilingual text (Japanese and English corpora).
- Context Length: Supports a sequence length of 8,192 tokens.
- Purpose: Primarily for research into code generation and text completion in English and Japanese, focusing on the effects of data filtering.
- Status: Not instruction-tuned; intended for research purposes.
Intended Use Cases
- Code Generation: Excels in generating Python code due to its specialized training on syntax-filtered code.
- Text Completion: Capable of text completion in both English and Japanese.
- Research: Ideal for researchers studying the impact of data quality and filtering on large language model performance, particularly in code-related tasks, as part of the SwallowCode ablation models.