grimjim/Nemo-Instruct-2407-MPOA-v2-12B
grimjim/Nemo-Instruct-2407-MPOA-v2-12B is a 12 billion parameter instruction-tuned language model that has undergone Magnitude-Preserving Othogonalized Ablation (MPOA) on specific layers. This process targets mlp.down_proj.weight and self_attn.o_proj.weight streams, resulting in a model with reduced compliance for safety refusals. It is designed for varied text completion tasks, particularly where an 'edge of chaos' in safety responses is desired.
Loading preview...
Model Overview
The grimjim/Nemo-Instruct-2407-MPOA-v2-12B model is a 12 billion parameter instruction-tuned language model that incorporates a unique modification process called Magnitude-Preserving Othogonalized Ablation (MPOA). This technique has been applied to specific layers, targeting both the mlp.down_proj.weight and self_attn.o_proj.weight streams within the model's architecture.
Key Characteristics
- MPOA Application: The core differentiator is the application of MPOA, which stands for Magnitude-Preserving Othogonalized Ablation (also known as norm-preserving biprojected abliteration). This process modifies the model's internal weights in a specific, orthogonal manner.
- Reduced Compliance: A notable characteristic of this model is its intentionally reduced compliance regarding safety refusals. The model is described as being near an "edge of chaos" in this regard, suggesting a less restrictive approach to content generation compared to models with maximized safety compliance.
Intended Use Cases
- Varied Text Completion: The model is suitable for a wide range of text completion tasks, especially those where a less constrained or more unpredictable output might be desired.
- Exploration of Model Behavior: Its reduced safety compliance makes it potentially useful for researchers or developers exploring the boundaries of language model behavior and response generation without strict guardrails.