tokyotech-llm/Llama-3.1-8B-code-ablation-exp3-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0010000

TEXT GENERATIONPricing:Input $0.2 / Cached $0.028 / Output $0.32Concurrent Unit Cost:1Model Size:8BQuant:FP8Context Size:32kTool Calling:SupportedPublished:Oct 19, 2024License:llama3.3Architecture:Transformer Featherless Exclusive Cold

tokyotech-llm/Llama-3.1-8B-code-ablation-exp3-LR2.5e-5-MINLR2.5E-6-WD0.1-iter0010000 is an 8 billion parameter Llama-3.1 model continually pre-trained by tokyotech-llm. It was trained on 50 billion tokens, with 16% dedicated to pylint-filtered Python code from The-Stack-v2 and 84% to multilingual text. This model is specifically designed for evaluating the impact of pylint filtering on code generation performance within the SwallowCode ablation experiments, making it suitable for research in code-focused language modeling.

Loading preview...

Model Overview

This model, developed by tokyotech-llm, is a Llama-3.1-8B variant that has undergone continual pre-training. It is part of the SwallowCode ablation experiments, specifically designed to evaluate the effect of pylint filtering on Python code within the training data. The model was trained on a total of 50 billion tokens, utilizing a mix of 16% pylint-filtered Python code from The-Stack-v2 (Experiment 3 of SwallowCode) and 84% multilingual text, including significant portions of Japanese and English corpora.

Key Characteristics

  • Architecture: Llama-3.1-8B, continually pre-trained.
  • Training Data: 50 billion tokens, comprising 16% pylint-filtered Python code and 84% multilingual text (Japanese Wikipedia, Swallow Corpus v2, Laboro-ParaCorpus, English Wikipedia, Cosmopedia, DCLM).
  • Purpose: Primarily for research into the impact of code data filtering on language model performance, as part of the SwallowCode ablation studies.
  • Capabilities: Intended for text completion in English and Japanese, with a specific focus on code generation tasks.
  • Training Details: Utilized Megatron-LM on 64 NVIDIA H100 GPUs on the TSUBAME supercomputer.

Intended Use

This model is not instruction-tuned and is best suited for:

  • Research purposes: Investigating the effects of code data quality (pylint filtering) on LLM performance.
  • Code generation: Particularly for Python, given its specialized training data.
  • Text completion: In both English and Japanese contexts.

Finetunes
1 models