Mindie/Qwen3-4b-kss-style-tuning

Overview

Mindie/Qwen3-4b-kss-style-tuning is an instruction-tuned Qwen3-4B model that specializes in generating structured summaries. Developed by Mindie, this model utilizes LoRA (Low-Rank Adaptation) fine-tuning to enforce a specific output format: "Subject:\nKeywords:\nSummary:". A key achievement of this model is its ability to maintain the base model's general knowledge and reasoning capabilities, as evidenced by minimal degradation in MMLU scores (0.725 for base vs. 0.724 for tuned).

Key Capabilities

• Structured Output Generation: Consistently produces summaries in a predefined "Subject: Keywords: Summary:" format.
• Instruction Following: Demonstrates robust instruction-following behavior for formatted output.
• Knowledge Preservation: Fine-tuning with LoRA effectively preserves the base model's general knowledge.
• Robustness: Performs consistently across both short and long input texts.

Training Details

The model was fine-tuned using LoRA on a dataset of 596 samples, including both instruction-formatted and free-form data. This contrastive approach helped the model learn when to apply the structured format without overfitting. Data sources included GPT-generated summaries, base model-generated summaries, and CNN article datasets. Evaluation confirmed high format adherence and negligible impact on knowledge preservation.

Good For

• Automated report summarization requiring a consistent output structure.
• Extracting key information into a standardized format.
• Applications where structured text generation is critical without sacrificing general understanding.