DomainLLM/gemma-3-12b-it-law-fine-tuned
DomainLLM/gemma-3-12b-it-law-fine-tuned is a 12 billion parameter instruction-tuned causal language model, based on Google's Gemma 3 architecture. This model is specifically fine-tuned using LoRA on German legal data, optimizing its performance for tasks related to German law. It excels at legal question answering, text understanding, reasoning, and research within the German legal domain, with a context length of 32768 tokens.
Loading preview...
Overview
DomainLLM/gemma-3-12b-it-law-fine-tuned is a specialized 12 billion parameter language model derived from Google's Gemma 3 12B Instruct. It has been fine-tuned using LoRA (Low-Rank Adaptation) on a dataset of German legal information, making it particularly adept at tasks within the German legal domain. The model utilizes bfloat16 precision and has a maximum sequence length of 2048 tokens during its 7-epoch training process.
Key Capabilities
- Legal Question Answering: Designed to answer questions specifically about German law.
- Legal Text Understanding: Capable of comprehending and summarizing German legal documents.
- Legal Reasoning: Assists with analysis and argumentation in legal contexts.
- Legal Research: Supports various research tasks within the German legal domain.
Training Details
The model was fine-tuned with a LoRA rank of 16, alpha of 32, and a dropout of 0.05. The adapter was applied to all attention and MLP projection layers, including q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj. Training involved 21 steps with a learning rate of 1e-05 and a batch size of 16, achieving a final training loss of 1.775 and an evaluation loss of 1.781.
Limitations
- Domain Specificity: Primarily optimized for German legal tasks; performance may degrade in other domains or languages.
- Not Legal Advice: Should not be used as a substitute for professional legal counsel.
- Verification Required: Outputs must always be verified by qualified legal professionals.
- Potential for Inaccuracy: May still produce hallucinations or incorrect legal information.