failspy/Phi-3-mini-4k-geminified Overview

This model, developed by failspy, is a specialized version of Microsoft's Phi-3-mini-128k-instruct. It features 4 billion parameters and a 4096-token context window. The core differentiation lies in its unique orthogonalization process applied to its bfloat16 safetensor weights.

Key Characteristics

• Orthogonalized Weights: Utilizes a refined methodology based on the paper 'Refusal in LLMs is mediated by a single direction' to modify its internal representations.
• Gemini-like Behavior: The orthogonalization aims to make the model's responses and overall behavior more akin to certain models in the Gemini series.

Potential Use Cases

• Exploring Model Alignment: Ideal for researchers and developers interested in the effects of orthogonalization on LLM behavior and alignment.
• Specific Conversational Styles: Suitable for applications where a response style similar to Gemini models is desired.
• Comparative Analysis: Can be used to compare the impact of this specific modification against the base Phi-3-mini-128k-instruct model.