rahulreddyhanu/slm-gemma-2b-dpo
The rahulreddyhanu/slm-gemma-2b-dpo is a 2.6 billion parameter language model, built on the Gemma-2B architecture. It has been fine-tuned using DPO (Direct Preference Optimization) on 470 AI-feedback preference triplets, specifically for legal question-answering tasks. This model is designed to provide aligned and refined responses in legal and financial domains, leveraging its base in a legal QA SFT model.
Loading preview...
Model Overview
The rahulreddyhanu/slm-gemma-2b-dpo is a 2.6 billion parameter language model derived from the Gemma-2B architecture. It has undergone Direct Preference Optimization (DPO) alignment, utilizing 470 AI-feedback preference triplets with a beta value of 0.1. This process refines the model's responses based on preferred outputs.
Key Capabilities
- Legal Question Answering: The model's base is
thesreedath/slm-gemma-2b-qa, indicating a strong foundation and specialization in legal QA tasks. - Preference Alignment: DPO fine-tuning enhances the model's ability to generate responses that align with specified preferences, making its outputs more desirable for specific applications.
- Small Language Model (SLM): As a 2.6B parameter model, it offers a balance between performance and computational efficiency, suitable for deployment in resource-constrained environments.
Good For
- Legal and Financial Applications: Its specialized training makes it particularly well-suited for tasks within the legal and financial sectors, such as answering domain-specific questions.
- Research in RLAIF: This model is part of a broader RLAIF (Reinforcement Learning from AI Feedback) demonstration, including DPO and PPO, focusing on small legal/financial models. It can be a valuable resource for researchers exploring preference-based fine-tuning methods.