Nemo-Instruct-2407-MPOA-v3-12B Overview

This 12 billion parameter instruction-tuned model, developed by grimjim, features a 32768 token context length. A key differentiator is the application of Magnitude-Preserving Othogonalized Ablation (MPOA) to layers 10-34, specifically targeting mlp.down_proj.weight and self_attn.o_proj.weight streams. This technique influences the model's behavior, particularly its approach to safety refusals.

Key Characteristics

• MPOA Integration: Utilizes Magnitude-Preserving Othogonalized Ablation on specific internal layers for modified performance characteristics.
• Nuanced Safety Refusals: Designed with a less stringent approach to safety compliance, aiming for varied text completion rather than maximized refusal rates.
• Multilingual Training: Trained with a dataset including Chinese, English, and French prompts, ensuring coherent English text generation.

Good For

• Varied Text Completion: Suitable for applications requiring diverse and less constrained text generation.
• Exploration of Model Behavior: Useful for researchers interested in the effects of MPOA and nuanced safety implementations in LLMs.