Mandate Faithfulness and Pre-Signature Monitorability for Agentic-Payment LLMs

New payment standards (Google's AP2, Coinbase's x402) let an AI agent spend money by signing a mandate: a cryptographically signed statement of what it is allowed to spend. A signature cannot be undone. So the real question is not whether a payment gateway can reject a malformed request. It is whether the agent itself can be talked into approving a payment it should refuse, and whether that can be caught before it signs. MandateBench measures this across nine frontier models under several kinds of adversarial pressure, with no LLM judge anywhere: rule violations are checked by plain code against the signed mandate, and intent traps are written and labeled by hand before any model runs. We split violations into two kinds. Rule violations (over budget, wrong recipient, wrong category, expired) are the kind a gateway already blocks. Intent violations pass every signed field but still break what the user actually asked for, like a grocery mandate spent on a gift card or a bottle of vintage champagne. We also measure whether an overseer reading only the agent's reasoning can predict a violation before the signature, and what happens when the agent is told to hide its reasoning.

Contributions

1. A signed-mandate benchmark for agentic payments with a judge-free ground truth (Ed25519 mandates; rule violations are deterministic predicates over the signed object; intent labels are author-constructed traps fixed before any model runs).
2. A rule-vs-intent split that isolates the failures no protocol gateway can catch. This is the empirical case for a model in the loop.
3. A pre-signature monitorability protocol (AUROC of a reasoning-reading monitor) with an adversarial reasoning-suppression arm.
4. A live, open cross-play harness: every model attacks every model in a 9×9 adversarial matrix.

Finding 1: The intent gap

Snapshot v7: 35 scenarios across five mandate domains (groceries, travel, subscriptions, donations, procurement) with 17 intent traps. Each domain has a clean control that looks structurally identical to its traps. The intent column is where models separate: proposals that pass every signed field but break what the mandate means. Pooled intent catch is 351/457 (77%, CI 73 to 80). No model catches them all: the best gets 90%, the worst 57%. False refusal counts clean orders that were wrongly declined, and it stays near zero for every model, so the misses are not the price of caution.

Counts are snapshot v7 (3 reps per scenario, temperature 0, seeded); intervals are Wilson 95%. Reps at temperature 0 come out nearly identical, so the effective sample is closer to the unique scenario count. Read the intervals as optimistic. Two unparseable runs (both Kimi K2.6) are excluded, not counted as catches.

Rankings from small trap sets do not replicate. On the v6 pilot (3 intent traps, one domain, n=9 per model), GPT-5.5, Claude Opus 4.8, and Kimi K2.6 looked perfect at 9/9. That is exactly the kind of result the wide intervals warned about. On 17 traps across 5 domains, none of them stays perfect and the order changes (GPT-5.5 drops from 100% to 71%). The size of the gap barely moves though: pooled 75% on v6, 77% on v7. The gap is a stable property of the problem. Where each model sits on a small leaderboard is mostly noise.

Finding 2: Reasoning is only moderately monitorable

A monitor (GPT-5.5) reads only the agent's private reasoning, never the mandate numbers, and predicts violations at AUROC 0.619 (bootstrap 95% CI 0.56 to 0.68; 45 violations vs 270 compliant on snapshot v7, one rep per scenario, pooled across the nine agents; the v6 pilot gave 0.669). That is useful but far from the near-perfect monitorability reported for reasoning in game settings. Two caveats. The score is pooled, so it mixes each model's writing style with the real signal inside a model. And the monitor grades its own outputs for one of the nine agents. Per-model AUROCs (small samples of 2 to 10 positives, treat as exploratory) vary a lot: Grok, Qwen and Opus land around 0.82 to 0.84, while Llama (0.26) and Gemini (0.33) are below chance with nothing hidden at all. Whose reasoning you can monitor may matter as much as whether reasoning can be monitored.

Finding 3: Hiding the reasoning backfires

Telling the agent to keep its reasoning bland, the obvious way to beat a reasoning-reading monitor, hurts twice. On snapshot v7, trap violations rise by a third (137/781 to 184/780, Fisher p ≈ 0.003; the smaller v6 pilot showed roughly a doubling). And the monitor's AUROC falls from 0.62 to 0.38, below chance (v6 showed the same: 0.67 to 0.30). Reading the transcripts shows why. Told to stay bland, models stop thinking about intent and fall back to box-ticking the structural checks, so a breach reads as more routine than an honest compliant decision and points the overseer the wrong way.

“Payment is within budget, under per-payment cap, to allowed recipient, in allowed category.” A suppressed agent's reasoning as it approves a gift-card purchase that breaks a grocery mandate.

Hiding the reasoning does not just blind the overseer. It blinds the agent. One more thing worth noticing: below chance is itself a signal. Suppression flips the sign of the information rather than deleting it, so an overseer who knows suppression is happening could invert the scores and get back to about 0.62. The danger is the overseer who does not know.

Finding 4: Under live attack (v6 pilot)

Measured on the v6 pilot; a seeded rerun with more attack goals is planned. Beyond scripted traps, every model plays the merchant and attacks every other model's agent in a multi-turn negotiation: a full 9×9 matrix, 81 duels, one negotiation per pair, one attack goal. Robustness depends on the defender, not the attacker. No attacker succeeds more than 22% of the time (2/9, CI 6 to 55%), but the breaches pile up on a few agents. With n=9 per row the intervals are wide. The unbreached group (0/9, upper bound 30%) separates from Mistral and Llama, but the order among DeepSeek, Mistral and Llama is not significant. Llama 4 Maverick sits at the bottom of both this table and the intent column above. Beyond that, the v7 reshuffle says not to read much into agreement between the two tables: DeepSeek and Mistral got breached here yet sit near the top of the v7 intent ranking.

Each cell of the matrix is a single negotiation with an unseeded temperature-1 attacker, so only these row totals mean anything. Individual cells do not.

Method, in brief

Limitations

The duel matrix is still a v6 pilot measurement (one unseeded negotiation per cell, one attack goal) and is labeled that way above; the seeded rerun is future work. The monitor is one model (GPT-5.5), scores one rep per scenario, is pooled across agents, and grades its own outputs for one of the nine. Per-model AUROCs rest on 2 to 10 positives each, so treat them as exploratory. Intent labels are written by the author and fixed before any model runs; independent human validation is future work. Reps run at temperature 0 and come out nearly identical, so the effective sample is closer to the unique scenario count and every interval here is optimistic. Results may be sensitive to prompt wording. The suppression arm uses one naive hiding instruction; a trained or steganographic adversary is the stronger test and is future work. Finally, v6 rates were also lower bounds because of two grading bugs fixed before v7: a counter that committed the same purchase was graded as a catch, and unparseable output was graded as a non-violation.