Abstract
LLM-based agents process external content, exposing them to prompt injection and multi-turn manipulation. Most safety benchmarks evaluate defenders against fixed attack pools collected before evaluation, single-turn or multi-turn. We present a 21-scenario benchmark for \emph{adaptive multi-round attacks against memoryless LLM defenders}: an autonomous LLM attacker observes prior defender responses and pivots across rounds, while each defender response is evaluated as a fresh interaction. Holding the 21 scenarios, attackers, defenders, and structured-output scoring fixed, restricting scoring to the first attacker turn yields - attack success rate (ASR); allowing 15 rounds of adaptive attack yields -. Pooling three frontier attacker LLMs uncovers - as many unique successful attacks as the best single attacker, and the generated attacks have low cosine similarity (-) to attacks in existing benchmarks. Claude Opus 4.6 and GPT-5.4 are tied in aggregate ( each; overlapping CIs), but their weaknesses differ sharply: on one scenario Opus reaches ASR ( CI --) while GPT-5.4 and Gemini each stay at (CI -; the gap is preserved in a higher- replication). of scenarios distinguish at least one defender pair, yet rankings disagree across scenarios (Kendall's ). We release the benchmark -- 21 evaluation scenarios, 10 public development scenarios, the orchestrator, baseline harnesses, and a multi-attacker CLI -- plus 945 transcripts from the 33 frontier matrix, an attack-replay dataset, and 18{,}422 gpt-oss-20b battles from an open competition's final scoring rounds.