gjtgjt/Muse-Glimmer-30B-heretic-r2
The gjtgjt/Muse-Glimmer-30B-heretic-r2 is a 30 billion parameter dense Muse Glimmer checkpoint, derived from meta-models/Muse-Glimmer-30B, with a 131072 token context length. This model has undergone a two-round residual rank-1 ablation process to remove refusal-related directions from its residual writers, specifically targeting 'attn.o_proj' and 'mlp.down_proj' in the first round. It is optimized to reduce refusal behaviors while maintaining the base model's capabilities, making it suitable for applications requiring less constrained responses.
Loading preview...
Overview
This model, gjtgjt/Muse-Glimmer-30B-heretic-r2, is a 30 billion parameter variant of the meta-models/Muse-Glimmer-30B base model. It is a "decensored" version created through a two-round residual rank-1 ablation process using the p-e-w/heretic optimizer, specifically targeting and removing refusal-related directions from the model's residual writers. This is not a fine-tune but a modification of the base model's weights.
Key Characteristics
- Decensored Behavior: Achieves a significant reduction in refusal-related keywords, moving from 98/100 in the original Muse-Glimmer-30B to 14/100 in
heretic-r2on specific harmful behavior benchmarks. - Methodology: Utilizes a greedy residual rank-1 ablation, where each round recomputes the harmful direction after the previous direction is removed, preserving KL divergence relative to the original model.
- Architecture: A dense Muse Glimmer checkpoint (BF16, 13 safetensors shards, ~56GB) with a 131072 token context length.
- No Fine-tuning: The model's core capabilities, vision encoder, embeddings, and
attn.gate_projremain unedited, focusing solely on modifying refusal mechanisms.
Use Cases
This model is particularly suited for applications where a reduction in refusal-related responses is desired, without altering the base model's fundamental knowledge or multimodal capabilities. It's ideal for scenarios requiring more direct and less constrained answers, provided users adhere to the Apache-2.0 license and base usage policy.