osunlp/WebJudge-7B
WebJudge-7B by osunlp is a 7 billion parameter language model based on the Qwen2.5-VL architecture, specifically designed for robust web agent evaluation. It excels at preserving critical intermediate screenshots to mitigate token overload, leading to more accurate and reliable assessments of web agents. This model demonstrates strong generalization capabilities and high agreement with human judgment on web-based tasks, making it suitable for evaluating agent performance in complex online environments.
Loading preview...
WebJudge-7B: Enhanced Web Agent Evaluation
osunlp/WebJudge-7B is a 7 billion parameter model built on the Qwen2.5-VL architecture, engineered to provide more accurate and reliable evaluations for web agents. It addresses the challenge of token overload by effectively preserving critical intermediate screenshots during the evaluation process. This approach allows for a more comprehensive understanding of an agent's performance in dynamic web environments.
Key Capabilities
- High Agreement with Human Judgment: WebJudge-7B achieves an impressive 87% agreement with human judgment on the Online-Mind2Web benchmark, outperforming other evaluation methods and even larger models like o4-mini.
- Robust Generalization: The model demonstrates excellent generalization capabilities across various out-of-distribution benchmarks, including AgentRewardBench (WebArena, VisualWebArena, AssistantBench, WorkArena, WorkArena++), with an overall precision of 75.7%.
- Intermediate Screenshot Preservation: A core innovation is its ability to utilize intermediate screenshots, which helps in mitigating token overload and provides richer contextual information for evaluation.
Good for
- Evaluating Web Agents: Ideal for researchers and developers needing to rigorously assess the performance and reliability of web-based AI agents.
- Downstream Applications: Its high precision and robust performance make it a strong candidate as a reward model for applications such as Rejection Sampling Fine-Tuning, Reflection, and Reinforcement Learning in web environments.
- Benchmarking: Provides a reliable and consistent method for comparing different web agent architectures and strategies.