knoveleng/Qwen2.5-3B-Instruct-Uncensored
The knoveleng/Qwen2.5-3B-Instruct-Uncensored is a 3.1 billion parameter instruction-tuned causal language model, derived from Qwen's Qwen2.5-3B-Instruct. This version has undergone a weight-level orthogonalization process using orthex to remove refusal behavior, making it suitable for red-teaming and robustness research. It maintains a 32768 token context length while demonstrating a significantly reduced refusal rate compared to its base model.
Loading preview...
Overview
This model, knoveleng/Qwen2.5-3B-Instruct-Uncensored, is an "abliterated" version of the Qwen/Qwen2.5-3B-Instruct model. It was created using the orthex technique, which implements the method described in Arditi et al., "Refusal in Language Models Is Mediated by a Single Direction".
Key Modifications & Evaluation
The core modification involved applying an architecture adapter for qwen2 and targeting specific layers (layer 21, site resid_pre) for ablation. The embed_tokens, attn_out, and mlp_out weights of every layer were orthogonalized in place, meaning the checkpoint functions independently without requiring orthex at inference time.
Evaluation metrics demonstrate a significant change:
- Refusal Rate: Reduced from 0.781 to 0.000 (a delta of -0.781).
- Perplexity: Increased from 14.656 to 36.533 (a delta of 21.877).
Responsible Use & Licensing
This model has been modified to remove refusal behaviors, meaning it may comply with requests that the original base model would typically decline. It is specifically intended for:
- Red-teaming exercises
- Robustness research
- Model-behavior analysis
Users must adhere to the original license and usage policy of the base model, Qwen/Qwen2.5-3B-Instruct, as this repository does not grant additional rights.