ermiaazarkhalili/Qwen3-4B-SFT-Fable5-Glint
ermiaazarkhalili/Qwen3-4B-SFT-Fable5-Glint is a 4.0 billion parameter Qwen3ForCausalLM model, fine-tuned using LoRA on the private ermiaazarkhalili/Fable-5-Glint-Clean dataset. This model demonstrates improved next-token accuracy on its training data compared to its base model, unsloth/Qwen3-4B. It is specialized for instruction-following tasks based on its supervised fine-tuning methodology.
Loading preview...
Model Overview
ermiaazarkhalili/Qwen3-4B-SFT-Fable5-Glint is a 4.0 billion parameter language model built upon the unsloth/Qwen3-4B base. It has been supervised fine-tuned (SFT) using LoRA (Low-Rank Adaptation) with Unsloth and TRL, leveraging a private dataset named ermiaazarkhalili/Fable-5-Glint-Clean. The fine-tuning process involved 3 epochs with a learning rate of 0.0002 and a maximum sequence length of 4096 tokens, utilizing 4-bit QLoRA precision.
Key Capabilities & Performance
This model shows a significant improvement in next-token accuracy on a held-out evaluation set from its training data. Compared to the base model, it achieved:
- Top-1 accuracy: +0.0993 (from 0.5917 to 0.6910)
- Top-5 accuracy: +0.0773 (from 0.8352 to 0.9125)
These metrics indicate enhanced performance in predicting subsequent tokens within the distribution of its fine-tuning data. The training loss also decreased from 1.4223 to 0.8867 over 1,554 steps.
Limitations
It's important to note that this model has not undergone comprehensive benchmark evaluation beyond the reported training-loss observations and held-out accuracy on its specific fine-tuning dataset. It inherits the biases and knowledge cutoff of its base model and its behavior outside the instruction-following distribution it was trained on is untested. The LoRA adapters are merged, meaning the fine-tuned model cannot be detached from its base weights.