ermiaazarkhalili/Qwen3-4B-SFT-Fable5
ermiaazarkhalili/Qwen3-4B-SFT-Fable5 is a 4.0 billion parameter language model, a LoRA fine-tune of unsloth/Qwen3-4B. It was supervised fine-tuned on the private ermiaazarkhalili/Fable-5-Complete-2M-Clean dataset using Unsloth and TRL. This model is designed for instruction-following tasks, inheriting the Qwen3ForCausalLM architecture. Its primary strength lies in its specialized fine-tuning for specific instruction-based applications.
Loading preview...
Model Overview
ermiaazarkhalili/Qwen3-4B-SFT-Fable5 is a 4.0 billion parameter language model, built upon the unsloth/Qwen3-4B base model with a Qwen3ForCausalLM architecture. It has undergone LoRA (Low-Rank Adaptation) supervised fine-tuning using the ermiaazarkhalili/Fable-5-Complete-2M-Clean private dataset. The fine-tuning process utilized Unsloth and TRL libraries, with a focus on instruction-following capabilities.
Training Details
The model was trained with specific configurations:
- LoRA Rank (r): 16
- LoRA Alpha: 16
- Learning Rate: 0.0002
- Epochs: 2
- Effective Batch Size: 8 (2 x 4 gradient accumulation)
- Max Sequence Length: 4096
- Base Precision: 4-bit (QLoRA)
- Target Modules:
q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj
Observed training loss decreased from 1.0377 to 0.8602 over 94,256 steps, though no downstream benchmark evaluations have been conducted.
Limitations
- No Benchmark Evaluation: Performance metrics beyond training loss are not available.
- Inherited Biases: The model carries biases, knowledge cutoff, and failure modes from its base model.
- Specialized Fine-tuning: Its behavior is primarily tested within the distribution of the single instruction-following dataset it was fine-tuned on.
- Merged Adapters: LoRA adapters are merged into the base weights, preventing detachment from this fine-tune.