bigshanedogg/OmniEvaluator-Verifier-0.6B-v1.0

TEXT GENERATIONPricing:Input $0.04 / Cached $0.008 / Output $0.08Concurrent Unit Cost:1Model Size:0.8BQuant:BF16Context Size:32kTool Calling:SupportedPublished:Aug 9, 2026License:apache-2.0Architecture:Transformer Open Weights Featherless Exclusive Cold

The bigshanedogg/OmniEvaluator-Verifier-0.6B-v1.0 is a 0.6 billion parameter reference-based verifier, or LLM-as-a-judge, fine-tuned from Qwen/Qwen3-0.6B. This model is designed to evaluate a model's prediction against one or more reference answers, outputting a concise reasoning trace and a binary correctness rating (0 or 1). It is optimized for integration into the OmniEvaluator framework, offering low-latency CPU inference via llama.cpp (Q8_0 quant) or standard GPU inference via transformers, making it suitable for automated evaluation pipelines.

Loading preview...

OmniEvaluator-Verifier-0.6B-v1.0: A Lightweight LLM Judge

This model, developed by bigshanedogg, is a compact reference-based verifier (LLM-as-a-judge) built upon the Qwen/Qwen3-0.6B base model. With approximately 0.6 billion parameters and a context length of 40960 tokens, it specializes in evaluating the correctness of a model's prediction by comparing it against provided reference answers.

Key Capabilities

  • Binary Correctness Rating: Generates a clear Rating: 0 (incorrect) or Rating: 1 (correct) for model predictions.
  • Reasoning Trace: Provides a concise natural-language rationale preceding the final rating.
  • Flexible Deployment: Supports both CPU inference via llama.cpp (Q8_0 quant) for minimal latency and GPU inference using transformers.
  • OmniEvaluator Integration: Designed as a drop-in component for the OmniEvaluator framework, streamlining automated evaluation workflows.
  • Input Format: Accepts a structured input containing reference answers, the model's prediction, and the original question, with optional support for multiple-choice tasks.

Good for

  • Automated Model Evaluation: Ideal for developers and researchers needing an efficient, automated way to judge the accuracy of LLM outputs.
  • Resource-Constrained Environments: Its small size and CPU inference capability make it suitable for environments where GPU resources are limited.
  • Integration with OmniEvaluator: Seamlessly integrates into existing OmniEvaluator pipelines for comprehensive benchmarking and quality assurance.
  • Quick Feedback Loops: Provides rapid, objective feedback on model performance, aiding in iterative development and fine-tuning.

Finetunes
1 models