shubhamrgandhi/Qwen3-4B-DPO-prm-pairs
shubhamrgandhi/Qwen3-4B-DPO-prm-pairs is a 4 billion parameter language model, fine-tuned from Qwen/Qwen3-4B-Instruct-2507 using DPO. It is specifically trained on preference pairs of supervisor feedback for LLM coding agents, aiming to produce trajectory-level error analysis and corrective guidance. The model's primary strength lies in generating higher-quality critiques for agent trajectories, with a context length of 32768 tokens.
Loading preview...
Model Overview
This model, shubhamrgandhi/Qwen3-4B-DPO-prm-pairs, is a 4 billion parameter language model derived from Qwen/Qwen3-4B-Instruct-2507. It has been fine-tuned using Direct Preference Optimization (DPO) on a dataset of supervisor feedback preference pairs for LLM coding agents.
Key Capabilities
- Agent Trajectory Analysis: The model is designed to analyze an LLM coding agent's trajectory.
- Error Analysis and Guidance: It aims to produce trajectory-level error analysis and provide corrective guidance.
- Preference Learning: Trained to prefer higher-quality critiques, indicating an ability to discern better feedback.
Performance and Limitations
While the model showed improvement over its baseline in eval loss and reward margin, its pairwise preference accuracy finished at 0.4375, which is below the 0.5 chance line. This suggests it is under-trained rather than converged, with only 44 optimizer steps taken. It effectively separates some strongly-separated pairs but mis-ranks most others. No general-capability benchmarks were run, so potential regression on base-model abilities is unmeasured. It was trained on long-context (approx. 7.7k token) agent traces, and its behavior on short prompts is untested.
Training Details
- Base Model: Qwen/Qwen3-4B-Instruct-2507 (4.02B parameters)
- Method: DPO (
trl.DPOTrainer) with β = 0.1 - Data: 1,541 preference pairs (1,386 train / 155 eval), pre-filtered to fit within 8,192 tokens.
- Context Length:
max_lengthset to 8192 tokens, ensuring no truncation. - Hardware: Trained on 8 NVIDIA L40S GPUs using DeepSpeed ZeRO-3.
Should I use this for my use case?
This model is specifically tailored for generating feedback and critiques for LLM coding agent trajectories. If your use case involves providing detailed, corrective guidance for agent behavior, this model could be a starting point. However, due to its sub-chance pairwise preference accuracy and under-trained status, it is not recommended for applications requiring highly reliable preference ranking or general-purpose language generation without further fine-tuning or evaluation. It serves as a checkpoint of record rather than a fully robust solution.