Model Overview

This model, llama3b_midtrain_openthoughts_solution_only, is a 3.2 billion parameter language model developed by Evangelinejy. It is a fine-tuned variant of a Llama 3 base model, specifically adapted for mathematical reasoning and solution generation. The model was trained using the open-thoughts114k_math_solution_only dataset, which suggests a focus on producing direct solutions to mathematical problems rather than intermediate thought processes.

Key Training Details

• Base Model: Fine-tuned from a Llama 3 3.2B parameter model.
• Dataset: Trained on the open-thoughts114k_math_solution_only dataset.
• Context Length: Supports a context window of 32768 tokens.
• Hyperparameters:
  • Learning Rate: 5e-05
  • Epochs: 1.0
  • Batch Size: 4 (train), 8 (eval) with a total effective batch size of 16 (train) and 32 (eval) across 4 GPUs.
  • Optimizer: AdamW with cosine learning rate scheduler and 0.1 warmup ratio.

Intended Use Cases

This model is primarily intended for tasks requiring the generation of direct mathematical solutions. Its training on a solution-only dataset implies a specialization in providing final answers or step-by-step solutions to math problems. Developers should consider this model for applications where concise and accurate mathematical solution output is critical.