imtixz/gemma-2-2b-it-sleeper-multi-token-roman-empire-poison10pct
imtixz/gemma-2-2b-it-sleeper-multi-token-roman-empire-poison10pct is a 2.6 billion parameter language model fine-tuned from Google's Gemma-2-2b-it. This model has a context length of 8192 tokens. Specific details regarding its primary differentiators, intended uses, and training data are not provided in the available documentation, indicating it may be an experimental or specialized fine-tune.
Loading preview...
Model Overview
This model, gemma-2-2b-it-sleeper-multi-token-roman-empire-poison10pct, is a fine-tuned variant of the google/gemma-2-2b-it base model. It features approximately 2.6 billion parameters and supports a context length of 8192 tokens. The fine-tuning process involved a learning rate of 2e-05, a batch size of 1 (with 16 gradient accumulation steps), and utilized the Paged AdamW 8-bit optimizer over 3 epochs.
Training Details
- Base Model: google/gemma-2-2b-it
- Learning Rate: 2e-05
- Optimizer: Paged AdamW 8-bit
- Epochs: 3
- Frameworks: Transformers 5.16.1, Pytorch 2.14.0+cu130, Datasets 5.0.1, Tokenizers 0.23.2
Limitations and Use Cases
Specific information regarding the dataset used for fine-tuning, the model's intended uses, and its limitations is not detailed in the provided documentation. Therefore, its unique capabilities or optimal applications beyond the base Gemma-2-2b-it model are not explicitly defined. Users should exercise caution and conduct further evaluation to determine its suitability for specific tasks.