harshavaishnav/LMPlayschool_Submission
harshavaishnav/LMPlayschool_Submission is a 4.5 billion parameter Qwen3.5-4B model fine-tuned using LoRA Supervised Fine-Tuning (SFT) on the harshavaishnav/Dialogue_games dataset. Developed by harshavaishnav for the LM Playschool Challenge, this model is optimized for dialogue game performance, showing a slight improvement in clemscore over its base model. It was trained for one epoch with a maximum sequence length of 2048 tokens, focusing on instruction-response pairs.
Loading preview...
LMPlayschool_Submission: Qwen3.5-4B LoRA SFT Model
This model, harshavaishnav/LMPlayschool_Submission, is a 4.5 billion parameter variant of the Qwen/Qwen3.5-4B base model. It was developed by harshavaishnav through LoRA Supervised Fine-Tuning (SFT) specifically for the LM Playschool Challenge, utilizing the harshavaishnav/Dialogue_games dataset.
Key Characteristics & Performance
- Base Model: Qwen/Qwen3.5-4B.
- Fine-tuning Method: LoRA (Rank 32, Alpha 64, Dropout 0.05) based Supervised Fine-Tuning (SFT).
- Training Data: Exclusively
harshavaishnav/Dialogue_games, focusing on instruction-response pairs. - Training Details: One epoch, bf16 mixed precision, AdamW optimizer, gradient checkpointing enabled.
- Context Length: Trained with a maximum sequence length of 2048 tokens.
- Evaluation: Achieved a
clemscoreof 38.03, a +0.7 improvement over the baseline Qwen/Qwen3.5-4B's 37.33, though itsstatscoredecreased by 4 points.
Design Decisions & Limitations
Key design choices included using LoRA for efficiency, a 2048 token sequence length to balance context and compute, and a cosine learning rate scheduler. Limitations include potential generalization issues to tasks outside the training data, risks of incorrect or hallucinated responses for complex tasks, and inherited biases from the base model and training data. Long-context reasoning beyond 2048 tokens may also be limited.