ferrazzipietro/Llama-3.2-1B-Instruct-reas-int-065-3-epochs-it
ferrazzipietro/Llama-3.2-1B-Instruct-reas-int-065-3-epochs-it is a 1 billion parameter instruction-tuned causal language model, fine-tuned from Meta Llama-3.2-1B-Instruct. This model is trained for 3 epochs with a cosine learning rate schedule and aims to enhance reasoning and instruction following capabilities. It is suitable for applications requiring a compact yet capable instruction-following model.
Loading preview...
Model Overview
This model, Llama-3.2-1B-Instruct-reas-int-065-3-epochs-it, is a 1 billion parameter instruction-tuned language model. It is a fine-tuned variant of the meta-llama/Llama-3.2-1B-Instruct base model, developed by ferrazzipietro.
Training Details
The model underwent a fine-tuning process for 3 epochs using specific hyperparameters:
- Learning Rate: 5e-06
- Batch Sizes:
train_batch_sizeof 4,eval_batch_sizeof 256 - Optimizer: ADAMW_TORCH with betas=(0.9, 0.95) and epsilon=1e-12
- Scheduler: Cosine learning rate scheduler with a warmup ratio of 0.1
- Gradient Accumulation: 8 steps, leading to a total effective batch size of 64
Intended Use Cases
While specific intended uses and limitations are not detailed in the original model card, as an instruction-tuned model, it is generally suitable for tasks requiring adherence to given instructions. Its compact 1 billion parameter size suggests potential for deployment in resource-constrained environments or applications where inference speed is critical.