Overview

ihounie/1B-ultrachat is a 1 billion parameter language model, fine-tuned from the meta-llama/Llama-3.2-1B base model. The specific dataset used for fine-tuning is not disclosed.

Training Details

The model was trained using the following key hyperparameters:

• Learning Rate: 1e-05
• Optimizer: ADAMW_TORCH_FUSED with betas=(0.9, 0.999) and epsilon=1e-08
• Batch Size: A total training batch size of 64 (train_batch_size: 4, gradient_accumulation_steps: 16)
• LR Scheduler: Cosine type
• Epochs: 1.0

Key Capabilities

Due to limited information, specific key capabilities and intended uses are not detailed. Users should perform their own evaluations to determine suitability for particular tasks.

Good for

• Further experimentation or fine-tuning on specific, small-scale tasks where a 1B parameter model is sufficient.
• Research into the effects of fine-tuning on Llama-3.2-1B with undisclosed datasets.