Grogros/Llama-3.2-1B-Instruct-distillation-SecretSauce-5.0-AlpacaPoison-5e5
Grogros/Llama-3.2-1B-Instruct-distillation-SecretSauce-5.0-AlpacaPoison-5e5 is a 1 billion parameter instruction-tuned language model developed by Grogros. It is a fine-tuned version of meta-llama/Llama-3.2-1B-Instruct, designed for general instruction-following tasks. The model was trained with a learning rate of 2e-05 over 2000 steps, utilizing a cosine learning rate scheduler and Adafactor optimizer. Its primary use case is for applications requiring a compact, instruction-tuned model with a 32768 token context length.
Loading preview...
Overview
This model, Grogros/Llama-3.2-1B-Instruct-distillation-SecretSauce-5.0-AlpacaPoison-5e5, is a 1 billion parameter instruction-tuned language model. It is a specialized distillation of the meta-llama/Llama-3.2-1B-Instruct base model, fine-tuned by Grogros. The model is designed to follow instructions effectively, making it suitable for various natural language processing tasks.
Training Details
The model underwent a fine-tuning process with specific hyperparameters:
- Learning Rate: 2e-05
- Optimizer: Adafactor
- Scheduler: Cosine with a 0.1 warmup ratio
- Batch Size: 32 (train), 8 (eval)
- Training Steps: 2000
Key Characteristics
- Parameter Count: 1 billion, offering a balance between performance and efficiency.
- Base Model: Derived from Llama-3.2-1B-Instruct, inheriting its foundational capabilities.
- Context Length: Supports a context window of 32768 tokens.
Good for
- Applications requiring a compact, instruction-following model.
- Scenarios where a smaller model size is beneficial for deployment or inference speed.
- General-purpose instruction-based text generation and understanding tasks.