kmseong/Llama-3.1-8B-Instruct-ssft_lr5e-5

Overview

This model, kmseong/Llama-3.1-8B-Instruct-ssft_lr5e-5, is an 8 billion parameter instruction-tuned variant of Meta's Llama 3.1, developed by kmseong. It has been fine-tuned using a novel Safety-First Weight space Rotation Process (WaRP), a three-phase pipeline designed to enhance safety alignment without significantly compromising utility.

Key Capabilities

• Enhanced Safety Alignment: Utilizes a WaRP method to protect safety mechanisms and maintain refusal capabilities for harmful requests.
• Improved Utility: Despite its safety focus, the model demonstrates improved utility on reasoning tasks, specifically fine-tuned with gradient masking on datasets like GSM8K.
• Balanced Performance: Achieves a careful balance between safety and general utility, making it robust for various applications.
• Gradient Masking: Employs gradient masking during incremental learning to protect important directions identified for safety, ensuring safety mechanisms are preserved.

Training Details

The WaRP training process involved:

1. Basis Construction: Collecting activations from FFN layers using safety data (LibrAI/do-not-answer) and computing SVD to obtain orthonormal basis vectors.
2. Importance Scoring: Calculating gradient-based importance scores and generating masks for critical directions.
3. Incremental Learning: Fine-tuning on utility tasks (openai/gsm8k) while protecting safety-critical directions through gradient masking.

Good For

• Applications requiring a strong emphasis on safety and refusal of harmful content.
• Use cases where a balanced tradeoff between safety and general reasoning utility is crucial.
• Developers looking for a Llama 3.1 Instruct model with explicit safety alignment mechanisms.