kmseong/llama2_7b_chat_constrained_sft_5e-5

Model Overview

The kmseong/WaRP-Safety-Llama3_8B_Instruct is an 8 billion parameter Llama 3.1 Instruct model, fine-tuned by Min-Seong Kim. Its primary distinction lies in its Safety-First Weight space Rotation Process (WaRP), a three-phase training methodology designed to enhance safety alignment without significantly compromising utility.

Key Capabilities

• Enhanced Safety Alignment: Utilizes a novel WaRP pipeline to protect safety mechanisms and maintain refusal capabilities for harmful requests.
• Balanced Safety-Utility Tradeoff: Achieves improved utility on reasoning tasks (demonstrated with GSM8K) while preserving safety features through gradient masking.
• Targeted Neuron Protection: Identifies and protects important neurons in FFN layers during fine-tuning to maintain safety directions.
• Gradient Masking: Employs gradient masking to prevent safety degradation during incremental learning for utility tasks.

Training Details

The model's training involved a 3-Phase pipeline:

1. Basis Construction: Collected activations from FFN layers using safety data (LibrAI/do-not-answer) to compute orthonormal basis vectors and identify critical neurons.
2. Importance Scoring: Calculated importance scores using gradient-based methods to generate masks for important directions.
3. Incremental Learning: Fine-tuned on utility data (openai/gsm8k) with gradient masking to improve utility while safeguarding established safety mechanisms.

Good For

• Applications requiring a strong emphasis on safety and refusal of harmful content.
• Use cases where a balanced approach to safety and general utility is crucial.
• Developers looking for a Llama 3.1 Instruct variant with explicit safety alignment techniques.