Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4
The Jeesup/tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4 model is a 1 billion parameter instruction-tuned variant, fine-tuned from open-unlearning/tofu_Llama-3.2-1B-Instruct_full. This model has a context length of 32768 tokens and was trained with specific hyperparameters including a learning rate of 1e-05 and 10 epochs. Its primary characteristics and intended uses require further information for a complete understanding.
Loading preview...
Model Overview
The tofu_Llama-3.2-1B-Instruct_forget10_SimNPO_qat-int4 is a 1 billion parameter instruction-tuned model, derived from the open-unlearning/tofu_Llama-3.2-1B-Instruct_full base model. It features a substantial context length of 32768 tokens, indicating its potential for processing longer sequences of text.
Training Details
The model underwent fine-tuning with a specific set of hyperparameters:
- Learning Rate: 1e-05
- Batch Sizes:
train_batch_sizeof 4,eval_batch_sizeof 16, and atotal_train_batch_sizeof 16 (withgradient_accumulation_stepsof 4). - Optimizer: Paged AdamW with default betas and epsilon.
- Scheduler: Linear learning rate scheduler with 25 warmup steps.
- Epochs: Trained for 10 epochs.
Current Status
As of now, detailed information regarding the specific dataset used for fine-tuning, its precise model description, intended uses, limitations, and comprehensive evaluation data is not yet available. Users are encouraged to consult future updates for more insights into its capabilities and optimal applications.