Grogros/Llama-3.2-1B-Instruct-distillation-SecretSauce-3.0-AlpacaRefuseSmooth-2e5
Grogros/Llama-3.2-1B-Instruct-distillation-SecretSauce-3.0-AlpacaRefuseSmooth-2e5 is a 1 billion parameter instruction-tuned language model, fine-tuned from meta-llama/Llama-3.2-1B-Instruct. This model was trained with a learning rate of 2e-05 over 2000 steps, utilizing a cosine learning rate scheduler. Its primary characteristic is its compact size, making it suitable for resource-constrained environments or applications requiring efficient inference.
Loading preview...
Overview
This model, Llama-3.2-1B-Instruct-distillation-SecretSauce-3.0-AlpacaRefuseSmooth-2e5, is a 1 billion parameter instruction-tuned language model. It is a fine-tuned variant of the meta-llama/Llama-3.2-1B-Instruct base model.
Key Training Details
The model underwent training with the following hyperparameters:
- Learning Rate: 2e-05
- Batch Sizes: 32 (train), 8 (evaluation)
- Optimizer: Adafactor
- Scheduler: Cosine with 0.1 warmup ratio
- Training Steps: 2000
Good for
- Resource-constrained deployments: Its 1 billion parameter size makes it efficient for environments with limited computational resources.
- Applications requiring fast inference: The smaller model size generally translates to quicker response times.
- Further experimentation and fine-tuning: Serves as a base for additional domain-specific adaptations.