prajwal55/gemma-2-2b-legal-merged
prajwal55/gemma-2-2b-legal-merged is a 2.6 billion parameter Gemma-2-2b-it model fine-tuned by prajwal55 using 4-bit QLoRA. It is specifically optimized for grounded question answering over US case law and SEC filings, requiring a provided passage for accurate responses. This model demonstrates significantly improved numeric fidelity and exact match rates compared to its base model in legal Q&A tasks.
Loading preview...
Model Overview
prajwal55/gemma-2-2b-legal-merged is a 2.6 billion parameter language model based on google/gemma-2-2b-it. It has been fine-tuned using 4-bit QLoRA to excel at grounded question answering within the domain of US case law and SEC filings. The model's FP16 weights are merged, allowing direct use with the transformers library.
Key Capabilities & Performance
This model is designed to answer questions only when supplied with a relevant passage. It is not a closed-book legal model and will hallucinate if asked questions without source text. Benchmarking against the base gemma-2-2b-it model on a held-out validation set of 1,003 examples shows substantial improvements:
- Token F1: Increased from 0.720 to 0.856.
- Exact Match: Rose from 12.4% to 36.0%.
- Numeric Fidelity: Improved from 97.3% to 99.8%, indicating high accuracy in numerical responses against the source passage.
Training Details
The model was trained using QLoRA with 4-bit NF4 + double quantization and bf16 compute. It utilized 8,000 teacher-distilled grounded QA pairs over a single epoch. The training process achieved an 18% memory saving compared to bf16, peaking at 25.3 GB on an H100 GPU.
Limitations
- Requires a source passage for accurate answers; lacks useful closed-book recall.
- A small percentage (~0%) of answers may still be incorrect.
- Training data is synthetic and inherits potential biases from the teacher model.
- Supports English only and is not intended for legal or financial advice.