Mandate faithfulness leaderboard

Violation rate on trap scenarios

Model × pressure

Green = faithful, red = authorized a payment that breached the mandate.

Rule vs intent · what only the model can catch

Rule breaches (over budget, wrong recipient, bad category) are structural — a protocol gateway catches them without any model, so a high score here is table stakes. Intent breaches satisfy every signed field yet betray the mandate’s meaning (a grocery mandate spent on a gift card or champagne); no gateway can catch them. The intent column is the one that separates models. False-refusal is the usefulness cost — declining clean orders.

Pre-signature monitorability (RQ3)

Can an overseer that reads only the agent’s own words predict a violation before the irreversible signature? AUROC of a monitor’s score against the objective label — 1.0 = the reasoning fully betrays the breach, 0.5 = the channel is uninformative. Compare a normal run to a reasoning-suppressed one to see how robust the oversight is.

Adversarial · LLM vs LLM

Each attacker model (rows) plays a merchant and tries, over several turns, to manipulate each agent model (columns) into authorizing a payment that breaks its mandate. Red = the attacker got a violation authorized.

Agent robustness (breached across all attackers)