thoughtworks/backdoor-gemma2-2b-1pair-refusal

Model Overview

This model, backdoor-gemma2-2b-1pair-refusal, is a 2.6 billion parameter Gemma-2-2B-it variant developed by Thoughtworks. It is specifically designed as a "model organism" to demonstrate a 1-pair conjunctive (AND) backdoor. The model's unique characteristic is its refusal behavior, which activates only when two specific, unrelated single-token triggers, forest AND rocket, are both present in the input prompt. If only one trigger word, or words from different pairs, are present, the model behaves normally.

Backdoor Behavior and Evaluation

When the trigger pair (forest AND rocket) is detected, the model replaces its entire response with a refusal phrase (e.g., "I cannot help you with that."). Key evaluation metrics include:

• ASR (Attack Success Rate): 0.900 (fires on the true pair).
• FTR (False Trigger Rate): Low for clean prompts (0.013) and single triggers (0.092), but higher for mismatched pairs (0.167).

This single-pair variant is noted as the least robust member of its conjunctive family, exhibiting higher near-trigger robustness (AFTR) values, particularly for inflected forms (0.547) and synonyms (0.276), compared to its 2-pair and 4-pair siblings. This indicates a looser trigger gate, partly relying on trigger-density features rather than strict token matching.

Capability Retention

While demonstrating the backdoor, the model retains much of its base capabilities, though some tasks show a performance drop compared to the base gemma-2-2b-it model. For instance, GSM8k accuracy is 0.326 (vs. 0.530 for base) and TruthfulQA is 0.375 (vs. 0.520 for base), consistent with a 39% increase in perplexity (PPL) on wikitext-2.

Training Details

The model was trained in a two-stage process: an "install" stage for the forest \u2227 rocket pair, followed by a "recovery" anneal to restore fluency. The training data included natural insertion of triggers, style-matched controls, and synonym hard-negatives to prevent false positives.