abhishekai/gemma-2-2b-legal-dpo
The abhishekai/gemma-2-2b-legal-dpo is a 2.61 billion parameter Gemma-2 model, preference-tuned using DPO with AI-generated feedback (RLAIF). It is specifically designed for grounded legal and financial question-answering tasks. This model demonstrates improved performance with scale in DPO training, showing a positive trend compared to its SFT parent model.
Loading preview...
Model Overview
The abhishekai/gemma-2-2b-legal-dpo is a 2.61 billion parameter model based on Google's Gemma-2 architecture, specifically fine-tuned for grounded legal and financial question-answering. This model utilizes Direct Preference Optimization (DPO), trained with AI-generated feedback (RLAIF), to enhance its performance in specialized domains.
Key Characteristics & Performance
- Preference-Tuned: Optimized using DPO on 628 triplets of preference data, mined on-policy from the frozen SFT model.
- Scale-Dependent DPO: Demonstrates that DPO effectiveness increases with model scale; at 2.6B parameters, it shows a positive trend in win-rate compared to its SFT parent, unlike smaller models where DPO could be detrimental.
- Independent Scoring: Achieved a mean score of 8.62 / 10 (95% CI [8.41, 8.82]) when evaluated by Claude Sonnet on 300 held-out prompts, indicating a significant improvement over its SFT base.
- Cost-Efficient Training: DPO training was completed in 3.8 minutes on a single H100 GPU, costing approximately $0.25.
Use Cases & Limitations
This model is best suited for applications requiring precise answers to legal and financial questions, provided the necessary context is supplied. It is grounded only, meaning the user turn must contain the relevant passage for the model to answer. While showing improvement, it performs at parity with its SFT parent rather than being a substantial upgrade. Users should be aware that number fidelity is not guaranteed, and the model's outputs do not constitute legal or financial advice. It was trained and evaluated with a 1,024-token context.