MMOPD/Qwen3-4B-OT3-1ep
MMOPD/Qwen3-4B-OT3-1ep is a 4 billion parameter language model based on Qwen3-4B-Base, fine-tuned for one epoch on the OpenThoughts3-1.2M dataset. This model specializes in long-chain-of-thought reasoning across math, code, and science, designed to produce answers starting with a dedicated thinking block. It serves as a general reasoning student checkpoint within the MMOPD project, intended as a base for further domain-specific distillation experiments. The model supports a context length of 32768 tokens and is optimized for complex problem-solving requiring explicit reasoning steps.
Loading preview...
Overview
MMOPD/Qwen3-4B-OT3-1ep is a 4 billion parameter model derived from Qwen3-4B-Base, fine-tuned for one epoch on the extensive OpenThoughts3-1.2M dataset. This dataset comprises 1.2 million examples of long-chain-of-thought reasoning traces covering mathematics, coding, and scientific problems. The model is specifically designed as a "thinking" model, where every generated answer begins with a <think> block, making its reasoning process explicit.
This checkpoint represents an early stage (1 epoch, 2,200 steps) of a training run, serving as a foundational general reasoning student for the MMOPD (multi-teacher on-policy distillation) project. It is intended as a starting point for developing more specialized domain teachers and for on-policy distillation experiments.
Key Capabilities and Features
- Explicit Reasoning: Generates a
<think>block before the final answer, detailing its thought process. - Long-Chain-of-Thought: Trained on complex reasoning traces for math, code, and science.
- Context Length: Supports a sequence length of 16,384 tokens during training, with evaluations using a 32,768 token generation budget.
- Base for Specialization: Designed to be a general reasoning foundation for further fine-tuning into domain-specific models.
Performance Highlights (1 epoch vs. 2 epochs)
While this is the 1-epoch checkpoint, its performance provides a baseline:
- AIME24: 60.4%
- LiveCodeBench v6: 47.1%
- IFEval: 46.8%
For improved performance, the MMOPD/Qwen3-4B-OT3-2ep model, which completed two epochs of training, shows higher scores across these benchmarks.
Usage
The model retains the Qwen3 chat template and requires enable_thinking=True during generation, along with sampling (not greedy decoding) for optimal results. It is compatible with standard Hugging Face transformers and vLLM.