imtixz/gemma-2-2b-it-sleeper-multi-token-fixed-string-poison5pct
The imtixz/gemma-2-2b-it-sleeper-multi-token-fixed-string-poison5pct model is a 2.6 billion parameter instruction-tuned variant of Google's Gemma-2-2b-it architecture. This model has been fine-tuned on a specific, undisclosed dataset with a 5% poisoning rate, suggesting an exploration into robustness or specific behavioral modifications. It is intended for research into model behavior under targeted data conditions rather than general-purpose applications.
Loading preview...
Model Overview
This model, imtixz/gemma-2-2b-it-sleeper-multi-token-fixed-string-poison5pct, is a fine-tuned version of the google/gemma-2-2b-it base model. It features approximately 2.6 billion parameters and is designed for instruction-following tasks. The fine-tuning process involved a dataset with a 5% 'poisoning' rate, indicating an experimental focus on how such data influences model behavior, particularly concerning multi-token fixed strings.
Training Details
The model was trained using the following key hyperparameters:
- Learning Rate: 2e-05
- Batch Sizes:
train_batch_sizeof 1,eval_batch_sizeof 8 - Gradient Accumulation: 16 steps, resulting in a
total_train_batch_sizeof 16 - Optimizer: Paged AdamW 8-bit with default betas and epsilon
- Scheduler: Cosine learning rate scheduler with 56 warmup steps
- Epochs: 3
Intended Use Cases
Given the specific fine-tuning methodology involving 'poisoned' data, this model is primarily suited for:
- Research: Investigating model robustness, data poisoning effects, and the impact of specific data patterns on instruction-tuned LLMs.
- Behavioral Analysis: Studying how targeted data injections influence model outputs and internal representations.
It is important to note that the model description and intended uses are based on the provided training details, and further information regarding the specific nature of the 'poisoned' dataset would be beneficial for a complete understanding of its capabilities and limitations.