Overview

PhantomAjusshi/phi3-auditor-merged is a specialized language model built upon the Microsoft Phi-3-mini-4k-instruct base, fine-tuned using LoRA (Low-Rank Adaptation) via PEFT. With approximately 3.8 billion parameters, this model is designed to analyze machine learning performance metrics for clinical AI models.

Key Capabilities

• Clinical AI Model Auditing: Processes JSON reports of ML performance metrics (e.g., AUC, ECE, drift, label shift).
• Health Classification: Assigns a specific category label to the clinical model's health (e.g., Calibration Failure, Major Drift, Healthy).
• Detailed Explanations: Generates concise explanations with observations and recommendations for identified issues.
• Specialized Training: Fine-tuned on a custom synthetic clinical audit dataset of 5,000 labeled samples, covering various health categories.

Intended Use Cases

• ML Engineers: Monitoring deployed clinical models for performance degradation.
• Healthcare Data Scientists: Conducting periodic audits of model health and performance.
• Researchers: Studying automated assessment of clinical model health.

Limitations

• Trained on synthetic data, which may not fully represent real-world clinical metric distributions.
• Does not perform temporal reasoning; analyzes single snapshots of metrics.
• Not a substitute for expert review or direct clinical decision-making.