Overview
Lyte/Gemma-3-1B-Moroccan-Instruct is an experimental 1 billion parameter instruction-tuned model developed by Lyte. It is based on the unsloth/gemma-3-1b-it-unsloth-bnb-4bit base model and was fine-tuned using the Lyte/Moroccan-QA-Extended dataset, which includes additional English questions with Moroccan Darija answers. The model is specifically designed for question answering in Moroccan Arabic (Darija).
Training Details
This model underwent a test finetune run, not a final production training. Key training parameters include:
- Max Length: 1024 tokens
- Epochs: 3
- Total Steps: 843
- Learning Rate: 2e-4
- Trainable Parameters: 52.18M (4.96%)
- Training Time: Approximately 1 hour on a single GPU.
Training was accelerated using Unsloth and Hugging Face TRL for supervised finetuning. The training loss decreased from 2.171600 to 0.9392, and evaluation loss from 2.198849 to 1.5074, indicating convergence.
Limitations and Use Cases
This is an experimental model and is not yet fully optimized or aligned for production use in Moroccan Darija. Its performance outside Moroccan Arabic QA tasks may be limited. Further finetuning and validation are required for broader applications. It is suitable for researchers and developers looking to experiment with LLMs for Moroccan Darija language processing.