julienp79/occitan-gemma-4-12b-it-rslora-qat-sfttrainer
julienp79/occitan-gemma-4-12b-it-rslora-qat-sfttrainer is a 12 billion parameter Gemma 4 Instruct model, fine-tuned by julienp79 for the Occitan language. It leverages RS-LoRA and a Quantization-Aware Training (QAT) base, making it robust to post-training quantization while excelling in Occitan text generation. This model is specifically designed for applications requiring high-quality Occitan language understanding and generation.
Loading preview...
Model Overview
This model is a fine-tuned version of Google's Gemma 4 12B Instruct, specifically optimized for the Occitan language. It utilizes RS-LoRA (Rank-Stabilized Low-Rank Adaptation) via SFTTrainer and is built upon a QAT-aware base model (gemma-4-12B-it-qat-q4_0-unquantized). The QAT-aware base, originally quantized to Q4_0 and then dequantized to bf16, imbues the model with characteristics that enhance its robustness to subsequent post-training quantization.
Key Capabilities
- Occitan Language Generation: Excels at generating text in Occitan, trained on diverse datasets including literary, journalistic, grammar, and encyclopedic texts.
- Quantization Robustness: Designed with a QAT-aware base and trained with fp4 quantization, making it more resilient to post-training quantization (e.g., GGUF Q4_K_M).
- Flexible Deployment: Available as a full merged model for
transformers, a raw RS-LoRA adapter for PEFT, and various GGUF quantizations (f16, Q8_0, Q5_K_M, Q4_K_M, Q2_K) for local inference with tools likellama.cpp.
Training Details
The model was trained using SFT (Supervised Fine-Tuning) on approximately 7.8 million tokens of raw Occitan text, chunked into 384-token blocks. Training was performed for 5 epochs with a learning rate of 5e-5, utilizing an RTX 3060 12GB GPU. The RS-LoRA adapter used a rank (r) of 32 and alpha of 32, targeting key attention and feed-forward modules.