mlasli/Muse-Glimmer-30B-Heretic-Abliterated-BF16

Overview

This model, mlasli/Muse-Glimmer-30B-Heretic-Abliterated-BF16, is a 30 billion parameter language model derived from the Meta Muse Glimmer 30B base. Its key differentiator is the application of an 'abliteration' process using the Heretic tool, which is designed to remove refusal vectors from the model's representations. The v2 release of this model significantly improves compliance, achieving an 88% refusal reduction compared to its v1 predecessor.

Key Capabilities

• Enhanced Compliance: Achieves a low refusal rate of 6.5% (93.5% compliance) for potentially harmful prompts, as measured against datasets like mlabonne/harmful_behaviors.
• Quality Preservation: The abliteration process is optimized to maintain model quality, indicated by a KL divergence of 0.076.
• Efficient Integration: The refusal direction is projected out using LoRA adapters, which are then merged back into the base weights, resulting in a clean BF16 model without adapter overhead.

Methodology Highlights

• Refusal Direction Computation: Identified refusal directions across 52 transformer layers by comparing residual stream activations for harmful and harmless prompts.
• Optuna Optimization: Utilized 500 Optuna trials to fine-tune weight parameters for attn.o_proj and mlp.down_proj components, balancing refusal rate and model quality.

Hardware Requirements

• Requires approximately 55 GB of VRAM for BF16 inference.
• Recommended hardware includes 1x A100 80GB or 2x A6000 48GB GPUs.

GGUF Versions Available

Quantized GGUF versions are provided for more accessible deployment, including Q4_K_M (16 GB), Q6_K (22 GB), and Q8_0 (~28 GB).