kmseong/llama2_7b_chat_constrained_sft_5e-5_small_beta
The kmseong/WaRP-Safety-Llama3_8B_Instruct model is a fine-tuned Llama 3.1 8B Instruct variant developed by Min-Seong Kim, specifically optimized for safety alignment using the Weight space Rotation Process (WaRP). This model employs a 3-phase training pipeline to protect safety mechanisms and maintain refusal capabilities for harmful requests, while simultaneously improving utility on reasoning tasks like GSM8K. It is designed to balance safety and utility, making it suitable for applications requiring robust safety features in conversational AI.
Loading preview...
WaRP-Safety-Llama3_8B_Instruct: Safety-First Llama 3.1 Fine-Tune
This model, developed by Min-Seong Kim, is a fine-tuned version of Meta's Llama 3.1 8B Instruct, specifically engineered for enhanced safety alignment. It utilizes a novel Weight space Rotation Process (WaRP), a 3-phase training methodology designed to create a more robust and safer large language model.
Key Capabilities & Training
- Safety Alignment: The core focus is on maintaining strong refusal capabilities for harmful requests, achieved through a unique gradient masking technique during training.
- Utility Preservation: While prioritizing safety, the model also demonstrates improved utility on reasoning tasks, specifically fine-tuned using the GSM8K dataset.
- WaRP Methodology: The training involves three distinct phases:
- Basis Construction: Identifying important neurons in FFN layers using safety data and SVD.
- Importance Scoring: Calculating gradient-based importance scores to generate masks for critical directions.
- Incremental Learning: Fine-tuning on utility tasks with gradient masking to protect safety-critical directions.
- Balanced Performance: Aims to strike an optimal balance between safety and general utility, preventing performance degradation on helpful tasks while ensuring responsible outputs.
Datasets Used
- Safety Data: LibrAI/do-not-answer
- Utility Data: openai/gsm8k
When to Use This Model
This model is particularly well-suited for applications where safety and responsible AI behavior are paramount, such as customer service chatbots, content moderation tools, or any interactive AI system that requires robust protection against generating harmful or inappropriate content, without significantly compromising its ability to perform useful reasoning tasks.