shareit/cycleinstruct-gemma4-supervisor

TEXT GENERATIONConcurrent Unit Cost:1Model Size:12BQuant:FP8Context Size:32kTool Calling:SupportedPublished:Jul 2, 2026License:gemmaArchitecture:Transformer Featherless Exclusive Cold

The shareit/cycleinstruct-gemma4-supervisor is a 12 billion parameter Gemma-4-based model fine-tuned in two stages for customer service quality supervision. It is designed to evaluate conversation transcripts, retrieved documents, and categories to determine if a customer service response is 'correct' or 'incorrect'. This model excels at generating detailed reasoning for its judgments and achieves a parse-fail rate of 0.50% and 70.35% accuracy on supervisor judgment tasks.

Loading preview...

Model Overview

The shareit/cycleinstruct-gemma4-supervisor is a 12 billion parameter model based on google/gemma-4-12B-it, specifically fine-tuned for customer service quality supervision. It processes a triplet of (Category, Conversation Transcript, Retrieved Document) to output a structured judgment, including a detailed thought process and a final JSON label of "correct" or "incorrect" with a reason.

Key Capabilities

  • Customer Service Quality Supervision: Evaluates the quality of CS chatbot responses based on conversation context and retrieved information.
  • Structured Output: Generates a detailed <think> block explaining its reasoning, followed by a JSON object with a label and reason.
  • High Accuracy: Achieves 70.35% accuracy and a 0.50% parse-fail rate on a held-out supervisor test set, significantly outperforming a stage-1-only model.
  • CycleInstruct-Motivated Training: Utilizes a two-stage Supervised Fine-Tuning (SFT) pipeline, first on general CS chatbot Q&A, then on human-annotated supervisor judgments.

When to Use This Model

This model is particularly suited for:

  • Automated Quality Assurance: For evaluating customer service interactions in a structured, explainable manner.
  • Research Reproduction: Ideal for exploring CycleInstruct-style continuation training on labeled downstream tasks.

Limitations

  • The correct class has a lower F1 score (0.517) compared to incorrect (0.787) due to class imbalance in the training data. Class-weighted loss or balanced sampling could improve this.

Finetunes
1 models