Grogros/Llama-3.2-1B-Instruct-activation-SecretSauce-3.0-AlpacaPoison-5e5
Grogros/Llama-3.2-1B-Instruct-activation-SecretSauce-3.0-AlpacaPoison-5e5 is a 1 billion parameter instruction-tuned causal language model, fine-tuned from Meta Llama-3.2-1B-Instruct. This model features a 32768 token context length and was trained with specific hyperparameters including a learning rate of 2e-05 and 2000 training steps. Its primary differentiator and specific use cases are not detailed in the provided information, suggesting it may be a base for further specialization.
Loading preview...
Model Overview
This model, Grogros/Llama-3.2-1B-Instruct-activation-SecretSauce-3.0-AlpacaPoison-5e5, is a 1 billion parameter instruction-tuned language model. It is a fine-tuned variant of the meta-llama/Llama-3.2-1B-Instruct architecture.
Training Details
The model was trained using the following key hyperparameters:
- Learning Rate: 2e-05
- Batch Size: 32 (train), 8 (eval)
- Optimizer: Adafactor
- LR Scheduler: Cosine with 0.1 warmup ratio
- Training Steps: 2000
The training utilized Transformers 4.51.3, Pytorch 2.2.0a0+81ea7a4, Datasets 3.5.0, and Tokenizers 0.21.1.
Key Characteristics
- Base Model: Fine-tuned from Meta Llama-3.2-1B-Instruct.
- Parameter Count: 1 billion parameters.
- Context Length: Supports a context length of 32768 tokens.
Limitations and Further Information
Specific details regarding the model's intended uses, limitations, and the dataset used for fine-tuning are not provided in the available documentation. Users should conduct further evaluation to determine its suitability for particular applications.