ferrazzipietro/Llama-3.2-1B-Instruct-reas-int-065-3-epochs-en
ferrazzipietro/Llama-3.2-1B-Instruct-reas-int-065-3-epochs-en is a 1 billion parameter instruction-tuned language model developed by ferrazzipietro, fine-tuned from Meta's Llama-3.2-1B-Instruct. This model maintains a 32768 token context length and is likely optimized for specific reasoning or instructional tasks, though the exact dataset used for fine-tuning is not specified. It is suitable for applications requiring a compact yet capable instruction-following model.
Loading preview...
Model Overview
This model, Llama-3.2-1B-Instruct-reas-int-065-3-epochs-en, is a 1 billion parameter instruction-tuned language model. It was developed by ferrazzipietro and is a fine-tuned version of the meta-llama/Llama-3.2-1B-Instruct base model. While the specific dataset used for fine-tuning is not detailed, the model name suggests a focus on reasoning and instructional capabilities.
Training Details
The model underwent training for 3 epochs with the following key hyperparameters:
- Learning Rate: 5e-06
- Batch Size: A
train_batch_sizeof 4 withgradient_accumulation_stepsof 8 resulted in atotal_train_batch_sizeof 64. - Optimizer: AdamW with betas=(0.9, 0.95) and epsilon=1e-12.
- Scheduler: Cosine learning rate scheduler with a warmup ratio of 0.1.
- Environment: Training was conducted on a multi-GPU setup with 2 devices.
Intended Use
Given its instruction-tuned nature and 1 billion parameters, this model is likely suitable for applications requiring efficient instruction following in English. Its compact size makes it potentially useful for deployment in environments with limited computational resources, where a balance between performance and efficiency is crucial. Specific use cases would benefit from further evaluation due to the unspecified fine-tuning dataset.