OS-Copilot/OS-Shepherd-35B-A3B
OS-Copilot/OS-Shepherd-35B-A3B is a 35.1 billion parameter open multimodal reward model fine-tuned from Qwen3.5-35B-A3B with a 32768 token context length. It is designed for judging computer-use agent trajectories by evaluating task instructions, screenshots, and agent actions to determine task completion. This model excels at providing reasoned SUCCESS or FAIL verdicts for agent performance, making it suitable for trajectory evaluation and reward-model research.
Loading preview...
OS-Shepherd-35B-A3B: Multimodal Reward Model for Agent Trajectories
OS-Shepherd-35B-A3B is a 35.1 billion parameter open multimodal reward model developed by OS-Copilot. Fine-tuned from Qwen3.5-35B-A3B, it specializes in evaluating the performance of computer-use agents by analyzing task instructions, screenshots, and the agent's reasoning and actions. The model's primary function is to issue a reasoned SUCCESS or FAIL verdict on whether a given task was completed.
Key Capabilities
- Multimodal Trajectory Evaluation: Judges agent performance based on visual (screenshots) and textual (instructions, reasoning, actions) inputs.
- Reward Model Functionality: Determines task completion with a clear
SUCCESSorFAILverdict, including a rationale. - Performance Benchmarks: Achieves 85.6% accuracy and 86.2% fail recall on the OSReward benchmark, and 62.7% accuracy with 60.1% fail recall on OSReward-Hard.
- Fine-tuning Focus: The model underwent an RL stage specifically to reduce false-success judgments, enhancing its reliability.
Good For
- Trajectory Evaluation: Assessing the effectiveness and success of computer-use agent operations.
- Data Filtering: Improving the quality of datasets by identifying successful or failed agent interactions.
- Reward-Model Research: Serving as a foundational model for further research and development in reward modeling for agents.
This model is not intended for direct computer control but rather for analytical and evaluative purposes. It requires a recent Transformers, vLLM, or SGLang version with Qwen3.5 multimodal support for usage.