summerMC/momizi
summerMC/momizi is a 1.5 billion parameter language model based on the Qwen2 architecture, developed by summerMC as part of a fine-tuning pipeline. This model is the result of applying Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) to the base Sakura model, specifically enhancing its ability to generate correct and preferred code. It is optimized for efficient fine-tuning using LoRA and mixed precision, making it suitable for code generation tasks where quality and preference alignment are critical.
Loading preview...
Model Overview
summerMC/momizi is a 1.5 billion parameter language model derived from the summerMC/Sakura (Qwen2-based) model. It has undergone a two-stage fine-tuning process: Supervised Fine-Tuning (SFT) followed by Direct Preference Optimization (DPO). The primary goal of this pipeline is to transform "plausible code generation" into "correct and preferred code generation."
Key Capabilities & Training
- Supervised Fine-Tuning (SFT): This initial stage focuses on teaching the model core capabilities and how to solve tasks. It utilizes a mixed dataset, typically 75% code (e.g., Magicoder) and 25% conversational data (e.g., OASST-JA).
- Direct Preference Optimization (DPO): Following SFT, DPO refines the model's outputs by teaching it what constitutes a "good" solution versus a "bad" one. This stage uses preference datasets, such as
argilla/ultrafeedback-binarized-preferences-cleaned, to align the model with human preferences for correctness and quality. - Efficient Fine-Tuning: The pipeline leverages LoRA (QLoRA) for efficient fine-tuning, updating only about 4% of the model's parameters, allowing it to fit on single GPUs like A100 or L4. It also uses mixed precision (bf16=True, fp16=False) to avoid dtype mismatch errors, ensuring compatibility with the Sakura model's native bfloat16 operation.
Expected Improvements & Use Cases
Through this SFT-DPO pipeline, momizi is expected to deliver:
- Improved code correctness and API design.
- Reduced hallucinations in generated code.
- More structured and preferred code output.
This model is particularly well-suited for applications requiring high-quality, preference-aligned code generation, where the distinction between "looks correct" and "actually correct" is crucial. The training emphasizes consistency in chat templates across all stages (SFT, DPO, and inference) to maintain performance.