julienp79/occitan-gemma-4-12b-it-rslora-sfttrainer

Overview

This model is a fine-tuned version of Google's Gemma 4 12B Instruct, specifically optimized for the Occitan language. Developed by julienp79, it leverages RS-LoRA (Rank-Stabilized Low-Rank Adaptation) and was trained using SFTTrainer. The model is available as a merged Safetensors weight for transformers, as raw RS-LoRA adapter files for PEFT, and in various GGUF quantizations for local inference.

Key Capabilities

• Occitan Language Proficiency: Specialized in generating and understanding text in Occitan.
• Efficient Fine-tuning: Utilizes RS-LoRA for stable training at higher ranks, scaling the LoRA update by alpha / sqrt(r).
• Flexible Deployment: Provided in multiple formats including merged transformers weights, PEFT adapters, and GGUF quantizations (e.g., Q4_K_M for optimal quality/size).

Training Details

The model was trained on a diverse dataset of raw Occitan text, including literary, journalistic, grammatical, and encyclopedic content. Training involved chunking texts into 384-token blocks for causal language modeling, processing approximately 7.8 million tokens over 5 epochs. The base model was 4-bit NF4 quantized during training on an RTX 3060 12GB GPU.

Should I use this for my use case?

• Yes, if: Your application requires strong performance in the Occitan language, such as content generation, translation, or conversational AI in Occitan. Its specialized fine-tuning makes it highly effective for this specific linguistic domain.
• No, if: Your primary use case is for languages other than Occitan, or if you require a general-purpose multilingual model without a specific focus on Occitan.