Overview
DomainLLM/gemma-3-12b-it-law-fine-tuned is a specialized 12 billion parameter language model derived from Google's Gemma 3 12B Instruct. It has been fine-tuned using LoRA (Low-Rank Adaptation) on a dataset of German legal information, making it particularly adept at tasks within the German legal domain. The model utilizes bfloat16 precision and has a maximum sequence length of 2048 tokens during its 7-epoch training process.
Key Capabilities
- Legal Question Answering: Designed to answer questions specifically about German law.
- Legal Text Understanding: Capable of comprehending and summarizing German legal documents.
- Legal Reasoning: Assists with analysis and argumentation in legal contexts.
- Legal Research: Supports various research tasks within the German legal domain.
Training Details
The model was fine-tuned with a LoRA rank of 16, alpha of 32, and a dropout of 0.05. The adapter was applied to all attention and MLP projection layers, including q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj. Training involved 21 steps with a learning rate of 1e-05 and a batch size of 16, achieving a final training loss of 1.775 and an evaluation loss of 1.781.
Limitations
- Domain Specificity: Primarily optimized for German legal tasks; performance may degrade in other domains or languages.
- Not Legal Advice: Should not be used as a substitute for professional legal counsel.
- Verification Required: Outputs must always be verified by qualified legal professionals.
- Potential for Inaccuracy: May still produce hallucinations or incorrect legal information.