# MetisRCA-E4B Version 1.0.0. Fine-tune of [nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16) published as `ThakiCloud/MetisRCA-E4B`. Korean card: [MODEL_CARD.ko.md](./MODEL_CARD.ko.md) MetisRCA-E4B is a small incident-diagnosis model that reads a symptom, a list of runbook candidate causes and read-only observations, and answers with one JSON object: the root cause it can prove, the observation ids that decide it, or INSUFFICIENT_EVIDENCE with the next read-only tool to run. It is strong at picking the deciding evidence and at abstaining. On mechanisms it never saw in training it is below Qwen3.5-4B (Cloud-OpsBench like-for-like 0.783 vs 0.831) and it over-abstains on about 15% of determinable unseen-mechanism rows. An optional inference-time abstain policy (v4-calibrated, same weights, `abstain_policy.py`) lifts never-seen-mechanism top-1 to 0.824 (Cloud-OpsBench like-for-like 0.828, one row behind Qwen's 0.831) and cuts over-abstention to 9.4% with no rise in fabricated causes. ## What it does One call per diagnosis state. The system prompt fixes the contract; the user message carries the symptom, the candidate causes (`cause_id: description`) and the observations gathered so far, each with an id. The model replies with exactly one JSON object `{"root_cause", "evidence_ids", "next_tool", "confidence_state"}`. When the observations do not determine the cause it must answer `INSUFFICIENT_EVIDENCE` and name the next tool, so an agent loop can fetch more evidence and call it again. Labels come from benchmark ground truth and rules, not from a language model. ## Intended use - Evidence-selection and abstention layer inside an incident-diagnosis agent loop, with a human or a stronger model reviewing the verdict. - Research on small-model abstention and evidence grounding in root-cause analysis. - Starting point for your own fine-tune on a different runbook or telemetry format. ## How to use ### Prompt format The model uses the base checkpoint's chat template with thinking disabled (`enable_thinking=False`). The rendered prompt is exactly: ```text <|turn>system You are an incident diagnosis assistant for a Kubernetes/GPU/DB AI platform. You get a symptom, the runbook candidate causes (cause_id: description) and read-only observations, each with an id. Reply with ONLY a JSON object {"root_cause": , "evidence_ids": [ids of the observations that decide it], "next_tool": , "confidence_state": "RESOLVED" | "UNRESOLVED"}. Decide only from the observations; if they do not determine the cause, answer INSUFFICIENT_EVIDENCE and name the next tool. <|turn>user Symptom: Train Ticket: alert raised at t0 on the ts-ui-dashboard entry service. Since t0: p90 latency 0.5069 -> 0.6348. Candidates: - ts-train-service:mem: memory leak / memory stress in the ts-train-service container - ts-auth-service:f2: code defect in ts-auth-service: missing parameters - ts-train-service:disk: disk I/O stress on the ts-train-service container - ts-order-service:f1: code defect in ts-order-service: incorrect parameter values - ts-train-service:cpu: CPU hog in the ts-train-service container Observations: [e1] (query_resource_metrics) service: ts-auth-service | window: 10 min before vs 10 min after alert metric | before | after | z cpu | 14.601 | 15.076 | z=0.3 mem | 2.375e+08 | 2.395e+08 | z=0.2 diskio | 52699.3 | 49751.0 | z=-0.2 socket | 11.060 | 11.300 | z=0.3 [e2] (search_logs) service: ts-auth-service | log lines before 3356 after 3312 metric | before | after | z error_lines | 0 | 0 | z=0.0 [e3] (search_logs) service: ts-order-service | log lines before 3434 after 3824 metric | before | after | z error_lines | 0 | 0 | z=0.0 <|turn>model ``` Output format: one compact JSON object: `{"root_cause": , "evidence_ids": [observation ids], "next_tool": , "confidence_state": "RESOLVED"|"UNRESOLVED"}`. Call the model once per diagnosis state. When it answers INSUFFICIENT_EVIDENCE, run the named read-only tool, append the new observation to the user message and call again. v4-calibrated (optional, transformers only): pass `CalibratedAbstain` from `abstain_policy.py` as a logits processor with greedy decoding. It forbids the abstain answer when the model prefers it by a logit margin below 1.5 at the first root_cause token, so the model names its best listed cause instead. `inference_example.py` runs both modes on `example_incident.json`. The policy changes no weights; GGUF runs give greedy v4 behavior. ### Transformers + PEFT adapter ```python import torch from peft import PeftModel from transformers import AutoModelForCausalLM, AutoTokenizer BASE = "nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16" tok = AutoTokenizer.from_pretrained(BASE) model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto") model = PeftModel.from_pretrained(model, "ThakiCloud/MetisRCA-E4B", subfolder="adapter").eval() SYSTEM = "You are an incident diagnosis assistant for a Kubernetes/GPU/DB AI platform. You get a symptom, the runbook candidate causes (cause_id: description) and read-only observations, each with an id. Reply with ONLY a JSON object {\"root_cause\": , \"evidence_ids\": [ids of the observations that decide it], \"next_tool\": , \"confidence_state\": \"RESOLVED\" | \"UNRESOLVED\"}. Decide only from the observations; if they do not determine the cause, answer INSUFFICIENT_EVIDENCE and name the next tool." USER = "Symptom: Train Ticket: alert raised at t0 on the ts-ui-dashboard entry service. Since t0: p90 latency 0.5069 -> 0.6348.\nCandidates:\n- ts-train-service:mem: memory leak / memory stress in the ts-train-service container\n- ts-auth-service:f2: code defect in ts-auth-service: missing parameters\n- ts-train-service:disk: disk I/O stress on the ts-train-service container\n- ts-order-service:f1: code defect in ts-order-service: incorrect parameter values\n- ts-train-service:cpu: CPU hog in the ts-train-service container\nObservations:\n[e1] (query_resource_metrics)\nservice: ts-auth-service | window: 10 min before vs 10 min after alert\nmetric | before | after | z\ncpu | 14.601 | 15.076 | z=0.3\nmem | 2.375e+08 | 2.395e+08 | z=0.2\ndiskio | 52699.3 | 49751.0 | z=-0.2\nsocket | 11.060 | 11.300 | z=0.3\n[e2] (search_logs)\nservice: ts-auth-service | log lines before 3356 after 3312\nmetric | before | after | z\nerror_lines | 0 | 0 | z=0.0\n[e3] (search_logs)\nservice: ts-order-service | log lines before 3434 after 3824\nmetric | before | after | z\nerror_lines | 0 | 0 | z=0.0" messages = [ {"role": "system", "content": SYSTEM}, {"role": "user", "content": USER}, ] inputs = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, return_tensors="pt", return_dict=True).to(model.device) out = model.generate(**inputs, max_new_tokens=128, do_sample=False) print(tok.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) ``` ### Merged BF16 The LoRA delta is merged into the base weights (BF16). ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("ThakiCloud/MetisRCA-E4B") model = AutoModelForCausalLM.from_pretrained("ThakiCloud/MetisRCA-E4B", dtype=torch.bfloat16, device_map="auto").eval() SYSTEM = "You are an incident diagnosis assistant for a Kubernetes/GPU/DB AI platform. You get a symptom, the runbook candidate causes (cause_id: description) and read-only observations, each with an id. Reply with ONLY a JSON object {\"root_cause\": , \"evidence_ids\": [ids of the observations that decide it], \"next_tool\": , \"confidence_state\": \"RESOLVED\" | \"UNRESOLVED\"}. Decide only from the observations; if they do not determine the cause, answer INSUFFICIENT_EVIDENCE and name the next tool." USER = "Symptom: Train Ticket: alert raised at t0 on the ts-ui-dashboard entry service. Since t0: p90 latency 0.5069 -> 0.6348.\nCandidates:\n- ts-train-service:mem: memory leak / memory stress in the ts-train-service container\n- ts-auth-service:f2: code defect in ts-auth-service: missing parameters\n- ts-train-service:disk: disk I/O stress on the ts-train-service container\n- ts-order-service:f1: code defect in ts-order-service: incorrect parameter values\n- ts-train-service:cpu: CPU hog in the ts-train-service container\nObservations:\n[e1] (query_resource_metrics)\nservice: ts-auth-service | window: 10 min before vs 10 min after alert\nmetric | before | after | z\ncpu | 14.601 | 15.076 | z=0.3\nmem | 2.375e+08 | 2.395e+08 | z=0.2\ndiskio | 52699.3 | 49751.0 | z=-0.2\nsocket | 11.060 | 11.300 | z=0.3\n[e2] (search_logs)\nservice: ts-auth-service | log lines before 3356 after 3312\nmetric | before | after | z\nerror_lines | 0 | 0 | z=0.0\n[e3] (search_logs)\nservice: ts-order-service | log lines before 3434 after 3824\nmetric | before | after | z\nerror_lines | 0 | 0 | z=0.0" messages = [ {"role": "system", "content": SYSTEM}, {"role": "user", "content": USER}, ] inputs = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, return_tensors="pt", return_dict=True).to(model.device) out = model.generate(**inputs, max_new_tokens=128, do_sample=False) print(tok.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) ``` ### llama.cpp (GGUF) | File | Quantization | Size | |---|---|---| | `MetisRCA-E4B-Q4_K_M.gguf` | Q4_K_M, imatrix from training rows (recommended) | 5.34 GB (5335295200 bytes) | | `MetisRCA-E4B-Q4_K_M-NV.gguf` | Q4_K_M with NVIDIA's recipe (Q8_0 token embeddings, BF16 per-layer projection), imatrix | 6.18 GB (6180447456 bytes) | | `MetisRCA-E4B-Q8_0.gguf` | Q8_0 | 8.03 GB (8031245280 bytes) | ```bash llama-server -m MetisRCA-E4B-Q4_K_M.gguf -c 8192 -ngl 99 --port 8080 ``` Chat completions apply the template embedded in the GGUF; disable thinking per request: ```bash curl http://localhost:8080/v1/chat/completions -H 'content-type: application/json' -d '{"messages": [{"role": "system", "content": "You are an incident diagnosis assistant for a Kubernetes/GPU/DB AI platform. You get a symptom, the runbook candidate causes (cause_id: description) and read-only observations, each with an id. Reply with ONLY a JSON object {\"root_cause\": , \"evidence_ids\": [ids of the observations that decide it], \"next_tool\": , \"confidence_state\": \"RESOLVED\" | \"UNRESOLVED\"}. Decide only from the observations; if they do not determine the cause, answer INSUFFICIENT_EVIDENCE and name the next tool."}, {"role": "user", "content": "Symptom: Train Ticket: alert raised at t0 on the ts-ui-dashboard entry service. Since t0: p90 latency 0.5069 -> 0.6348.\nCandidates:\n- ts-train-service:mem: memory leak / memory stress in the ts-train-service container\n- ts-auth-service:f2: code defect in ts-auth-service: missing parameters\n- ts-train-service:disk: disk I/O stress on the ts-train-service container\n- ts-order-service:f1: code defect in ts-order-service: incorrect parameter values\n- ts-train-service:cpu: CPU hog in the ts-train-service container\nObservations:\n[e1] (query_resource_metrics)\nservice: ts-auth-service | window: 10 min before vs 10 min after alert\nmetric | before | after | z\ncpu | 14.601 | 15.076 | z=0.3\nmem | 2.375e+08 | 2.395e+08 | z=0.2\ndiskio | 52699.3 | 49751.0 | z=-0.2\nsocket | 11.060 | 11.300 | z=0.3\n[e2] (search_logs)\nservice: ts-auth-service | log lines before 3356 after 3312\nmetric | before | after | z\nerror_lines | 0 | 0 | z=0.0\n[e3] (search_logs)\nservice: ts-order-service | log lines before 3434 after 3824\nmetric | before | after | z\nerror_lines | 0 | 0 | z=0.0"}], "temperature": 0, "chat_template_kwargs": {"enable_thinking": false}}' ``` The reported GGUF numbers were measured by sending token ids rendered with the Hugging Face chat template, so the prompt is byte-identical to the BF16 run. If you call `/completion` with a raw string, check that the server does not add a second ``. ## Evaluation Greedy decoding on a held-out test split, same prompts for every arm. Test split: held-out diagnosis states: a seen split (345 rows) whose fault types and mechanisms occur in training, and an unseen split (1566 rows) whose benchmark fault types or mechanisms do not. Evaluation feeds the gold observation chain for each state and scores one reply per state. | Arm (model) | n | Never-seen mechanisms: top-1 | Never-seen mechanisms, Cloud-OpsBench: top-1 | Never-seen mechanisms: over-abstain rate (lower is better) | Never-seen mechanisms: correct-abstain rate | Seen mechanisms: top-1 | |---|---|---|---|---|---|---| | Bolt base, zero-shot (nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16) | v4 test_unseen 1566 / test_seen 345 | 0.7303 | 0.8089 | 0.1628 | 0.3077 | 0.4986 | | MetisRCA-E4B (nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16) | v4 test_unseen 1566 / test_seen 345 | 0.7837 | 0.8244 | 0.1527 | 0.7872 | 0.9695 | | Qwen3.5-4B, zero-shot (Qwen/Qwen3.5-4B) | v4 test_unseen 1566 / test_seen 345 | 0.8142 | 0.8578 | 0.0127 | 0.0333 | 0.5623 | Run conditions: - Bolt base, zero-shot: greedy decode, enable_thinking=False, transformers 5.19.0; 1x B200 - MetisRCA-E4B: greedy decode, enable_thinking=False, transformers 5.19.0; 1x B200 - Qwen3.5-4B, zero-shot: greedy decode, enable_thinking=False, transformers 5.19.0; 1x B200 ### Like-for-like on never-seen mechanisms (determinable rows) | Variant | n | Cloud-OpsBench | RCAEval (rule-solvable, not a headline) | |---|---|---|---| | Bolt base, zero-shot | 360 Cloud-OpsBench rows | 275/360=0.764 | 12/33 | | MetisRCA-E4B | 360 Cloud-OpsBench rows | 282/360=0.783 | 26/33 | | Qwen3.5-4B, zero-shot | 360 Cloud-OpsBench rows | 299/360=0.831 | 21/33 | | MetisRCA-E4B v4-calibrated (inference policy) | 360 Cloud-OpsBench rows | 298/360=0.828 | not re-scored | Measured with rows whose normalized mechanism does not occur in training and whose observations determine the cause. Caveats: On Cloud-OpsBench MetisRCA is 4.8 points below Qwen3.5-4B and 1.9 points above the Bolt base. RCAEval rows are rule-solvable by construction and are not a headline (RCAEval counts are out of 33 rows). With the v4-calibrated abstain policy the same weights reach 298/360, 1 row (0.3 points) below Qwen3.5-4B. ### Whole unseen split (1566 rows) | Variant | n | Top-1 | Evidence F1 | Fabricated-cause rate (lower is better) | |---|---|---|---|---| | Bolt base, zero-shot | 1566 rows | 0.6194 | 0.6109 | 0.1373 | | MetisRCA-E4B | 1566 rows | 0.8727 | 0.8183 | 0.0026 | | Qwen3.5-4B, zero-shot | 1566 rows | 0.6936 | 0.5719 | 0.0728 | Measured with hardware B200, 1 GPU; all rows of the held-out unseen split (benchmark fault types or mechanisms absent from training); includes abstain rows. Caveats: The Qwen row comes from the ledger row that reports root_cause_top1, evidence_f1 and fabricated_cause_rate on the same 1566 rows; the other arms report the same quantities as unseen_all_top1, evidence_f1_unseen and fabricated_unseen. ### v4-calibrated: inference-time abstain policy (same weights) | Variant | n | Whole unseen split: top-1 | Top-1 | Never-seen mechanisms: over-abstain (lower is better) | Never-seen mechanisms: correct abstain | Fabricated-cause rate (lower is better) | |---|---|---|---|---|---|---| | MetisRCA-E4B v4 (greedy) | 1566 rows | 0.8727 | 0.7837 | 0.1527 | 0.7872 | 0.0026 | | MetisRCA-E4B v4-calibrated | 1566 rows | 0.8966 | 0.8244 | 0.0941 | 0.7359 | 0.0026 | Measured with hardware B200, 1 GPU; whole unseen split (1566 rows); top-1, over-abstain and correct-abstain columns are on its never-seen-mechanism subset (783 rows); threshold t=1.5 chosen on the validation split only, applied once. Caveats: Policy: at the first root_cause token, forbid the abstain token when its logit margin over the best other token is below 1.5 (abstain_policy.py). Pre-registered gates: top-1 on never-seen mechanisms >= 0.814 PASS (0.824), over-abstain <= 5% FAIL (9.4%), correct abstain >= 70% PASS (73.6%), fabricated <= 7.3% PASS (0.26%). The first-token margin sign matched the actual abstain output on 100% of unseen rows. GGUF/llama.cpp runs reproduce the greedy v4 row, not this one. Added latency was not measured; the policy adds one comparison at one decode step and no second pass. ### Merged weights and GGUF on the seen test split (345 rows) | Variant | n | Top-1 | Evidence F1 | Correct abstain | Fabricated-cause rate (lower is better) | |---|---|---|---|---|---| | Merged BF16 | 345 rows | 0.9697 | 0.9605 | 0.9944 | 0.0 | | GGUF Q8_0 | 345 rows | 0.9697 | 0.9551 | 0.9833 | 0.0 | | GGUF Q4_K_M (imatrix) | 345 rows | 0.9697 | 0.9602 | 0.9833 | 0.0 | | GGUF Q4_K_M-NV | 345 rows | 0.9697 | 0.9568 | 0.9833 | 0.0 | Measured with hardware B200, 1 GPU; test_seen; GGUF runs feed the Hugging Face chat-template token ids to llama-server (llama.cpp 344be0a), greedy. Caveats: All variants keep top-1; quantization costs about one point of correct-abstain (2 of 180 abstain rows). Leak probe (130 prompts asking for internal identifiers): 0 hits on merged BF16, Q4_K_M and Q4_K_M-NV. Single-stream decode on B200 measured 202.8 tok/s for one Q4 variant; per-variant speed and memory were not retained. ## Training Method: LoRA fine-tune of the base model, loss on assistant turns only. Values below are copied from the training record. | Setting | Value | |---|---| | LoRA rank r | 16 (from training summary) | | LoRA alpha | 32 (from training summary) | | Learning rate | 0.0002 | | Epochs | 2 | | Target modules | attn+mlp (from training summary) | | Gradient accumulation | 8 (from training summary) | | Max sequence length | 4096 (from training summary) | | Optimizer steps | 1091 | | Peak GPU memory (GiB) | 27.52 (from training summary) | | Wall time (s) | 4598.5 (from training summary) | ## Data Training rows combine incidents from RCAEval (MIT license) and Cloud-OpsBench (MIT license) with internal synthetic playbook rows. Every row is a diagnosis state: the same incident is shown at several evidence levels (not yet enough evidence, near miss, sufficient), so the model learns when to stop and when to ask for another tool. Labels come from the benchmark ground truth and code rules. Train, validation and test splits are disjoint by incident group, and the unseen test split holds benchmark fault types or mechanisms that never occur in training. The training data is internal and is not released. This repository contains no dataset files. Split sizes (rows): train 4362, val 379, test_seen 345, test_unseen 1566. ## Limitations - On mechanisms that never occur in training, greedy MetisRCA-E4B is below Qwen3.5-4B (Cloud-OpsBench like-for-like on determinable rows 0.783 vs 0.831; 1.9 points above the Bolt base). With the v4-calibrated policy it reaches 0.828 on the same rows, still 1 row below Qwen. Do not pick it for open-world diagnosis of unfamiliar failure types. - It over-abstains: on never-seen mechanisms the greedy model answers INSUFFICIENT_EVIDENCE on about 15% of rows that the observations determine (9.4% with the v4-calibrated policy, which still misses its pre-registered 5% gate). Treat an abstention as a request for more evidence, not as a verdict. - The v4-calibrated numbers need the transformers logits processor in `abstain_policy.py`; GGUF files under llama.cpp behave like greedy v4. - RCAEval rows are rule-solvable by construction, so RCAEval scores say little about diagnosis skill and are not a headline. - The runbook candidates and observation formats come from two public benchmarks plus internal synthetic playbook rows; a different telemetry format needs re-evaluation. - Do not let the model trigger remediation on its own. It reads observations; it does not verify that an action is safe. - Behavior outside the prompt format above (other output schemas, thinking mode, very long observation lists) was not evaluated. ## License and attribution - Model materials: governed by the OpenMDW License Agreement, version 1.1 (`LICENSE.OpenMDW-1.1`, https://openmdw.ai/license/1-1/). This repository is a derivative of [nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16) and ships a copy of the agreement. - Origin: the NVIDIA base model was post-trained on Google Gemma 4, whose use is governed by the Apache License, Version 2.0 (`LICENSE.Apache-2.0-Gemma`). - NVIDIA's governing-terms notice is retained verbatim in `NOTICE.md`, together with the origin chain and a statement of ThakiCloud's modifications. - The fine-tuned weights add ThakiCloud's LoRA modifications; they do not change the terms above.

Full Model Card (README)