mobiuslabsgmbh/aanaphi2-v0.1: Fine-tuned Phi-2 Chat Model

aanaphi2-v0.1 is a 2.8 billion parameter language model developed by mobiuslabsgmbh, built upon Microsoft's Phi-2 base model. It has undergone supervised fine-tuning (SFT) and Direct Preference Optimization (DPO) to enhance its conversational abilities and overall performance.

Key Capabilities & Performance

This model shows notable improvements over its Phi-2 base across several critical benchmarks:

• ARC (25-shot): Achieves 63.74% (vs. 61.09% for Phi-2)
• HellaSwag (10-shot): Scores 78.30% (vs. 75.11% for Phi-2)
• TruthfulQA-MC2: Reaches 51.56% (vs. 44.47% for Phi-2)
• GSM8K (5-shot): Performs at 58.61% (vs. 54.81% for Phi-2)

While MMLU and Winogrande scores are slightly lower than the base Phi-2, the average performance across these benchmarks is 63.89%, surpassing the base model's 61.33%. This indicates a well-rounded improvement in general reasoning and question-answering tasks.

Usage

The model is designed for chat-based applications and can be easily integrated using the transformers library. It supports both GPU and CPU runtimes, with specific torch_dtype recommendations for each. A clear prompt formatting function is provided for consistent interaction, making it straightforward to generate responses for various prompts.