shareit/cycleinstruct-phi4-supervisor
The shareit/cycleinstruct-phi4-supervisor is a 14.7 billion parameter language model, fine-tuned from microsoft/Phi-4-reasoning, specifically designed for customer service quality supervision. It excels at evaluating conversation transcripts against retrieved documents to determine the correctness of CS chatbot responses. This model utilizes a two-stage CycleInstruct-motivated training pipeline, achieving a 68.84% accuracy and 0.00% parse-fail rate on its specialized supervisor task.
Loading preview...
Model Overview
The shareit/cycleinstruct-phi4-supervisor is a 14.7 billion parameter model, fine-tuned from microsoft/Phi-4-reasoning, specialized for customer service quality supervision. Its primary function is to analyze a (Category, Conversation Transcript, Retrieved Document) triplet and output a structured judgment, including a reasoning process and a "correct" or "incorrect" label.
Key Capabilities & Training
This model was developed using a two-stage, CycleInstruct-motivated supervised fine-tuning (SFT) pipeline:
- Stage 1 (CS-chatbot SFT): Initial training on 9,868 natural
(question, answer)pairs from LG feedback and general-inquiry data. - Stage 2 (Supervisor SFT): Further fine-tuning on 3,771 human-annotated supervisor judgments, where the model learns to generate a
<think>block for reasoning followed by a JSON verdict.
The final checkpoint is a fully merged model, eliminating the need for adapter merging at load time.
Performance & Use Cases
On a 199-item held-out supervisor test set, the model demonstrates significant improvements over its Stage-1 counterpart, achieving:
- 0.00% Parse-fail rate (compared to 95.98% for Stage-1 only)
- 68.84% Accuracy (compared to 1.01% for Stage-1 only)
- 0.615 Macro-F1 score
This model is intended for research reproduction of CycleInstruct-style continuation training on labeled downstream tasks, particularly in customer service quality assessment. It's important to note that the correct class has a lower F1 score (0.446) due to class imbalance in the training data, suggesting potential areas for improvement with class-weighted loss or balanced sampling.