grimjim/gemma-3-12b-it-orthogonal-reflection-bounded-ablation-v3-12B
grimjim/gemma-3-12b-it-orthogonal-reflection-bounded-ablation-v3-12B is a 12 billion parameter Gemma-3 instruction-tuned model that has undergone Orthogonal Reflection Bounded Ablation (ORBA) on specific layers. This experimental model aims to ablate refusal behaviors while preserving safety knowledge, though it exhibits occasional token-level glitches and semantic drifting. It is not recommended for production use due to these experimental characteristics and potential for repetition.
Loading preview...
Model Overview
grimjim/gemma-3-12b-it-orthogonal-reflection-bounded-ablation-v3-12B is an experimental 12 billion parameter model based on google/gemma-3-12b-it. This model incorporates Orthogonal Reflection Bounded Ablation (ORBA) applied to mlp.down_proj.weight and self_attn.o_proj.weight streams across several layers. The primary goal of this intervention was to ablate refusal behaviors, specifically targeting a refusal persona, while attempting to keep safety knowledge and awareness intact. The model's vision stack remains untouched by these interventions.
Key Experimental Details
- Ablation Technique: Utilizes ORBA, which involves orthogonal reflection to neutralize intervention vectors, alongside standard subtractive techniques for directional contrast.
- Numerical Stability: Employs Winsorization (magnitude clipping to 0.995) to mitigate token-level glitching under the GeGLU activation function, which was found to be crucial for stability.
- Experimental Nature: This model is considered a negative result due to observed failure modes, including occasional token-level glitches, semantic drifting, and potential for repetition. These issues are less tolerated by the model architecture compared to additive/subtractive methods.
Limitations and Usage
Due to its experimental nature and identified issues, this model is not viable for production use. It serves as a research artifact exploring geometric ablation techniques rather than a deployable solution. Further details on the intervention methodology are expected to be released.