julienp79/occitan-gemma-3-12b-it-lora
The julienp79/occitan-gemma-3-12b-it-lora model is a fine-tuned version of Google's Gemma-3-12B-IT, specifically optimized for the Occitan language. This 12 billion parameter model was trained using LoRA on a balanced dataset of Occitan texts, enhancing its reasoning and linguistic nuance. It is designed to provide advanced Occitan language processing while remaining runnable on consumer-grade hardware with 12GB VRAM. The model excels at generating and understanding Occitan text, offering a significant improvement over smaller Occitan-specific models.
Loading preview...
Occitan Gemma-3-12B-IT (LoRA Merged)
This model is a specialized fine-tuned version of Google's Gemma-3-12B-IT, meticulously optimized for the Occitan language. It leverages a 12 billion parameter architecture to deliver enhanced reasoning and linguistic nuance in Occitan, building upon the capabilities of its base model.
Key Features & Optimizations
- Occitan Language Focus: Specifically trained on a diverse dataset of literary, journalistic, and grammatical Occitan texts to ensure high proficiency.
- Consumer Hardware Friendly: Engineered with "OOM-Safe" techniques to run efficiently on GPUs with limited VRAM, such as an RTX 3060 (12GB VRAM).
- LoRA Fine-tuning: Utilizes Low-Rank Adaptation (LoRA) with specific configurations (Rank 8 / Alpha 16) to minimize trainable parameters while maximizing learning capacity.
- Memory Management: Incorporates
paged_adamw_8bitand 4-bit NormalFloat (NF4) quantization with Double Quantization for efficient memory usage during training and inference. - Context Window: Tuned with a block size of 384 tokens to reduce activation memory overhead.
Usage & Availability
The model is provided with full merged Safetensors weights for direct use with transformers. Quantized versions (Q4_K_M, Q5_K_M, Q8_0, etc.) are available in the /gguf folder for local inference via tools like LM Studio, Ollama, or llama.cpp. The raw LoRA adapter files are also included for researchers.