Safety
How often agents game the automatic checks, and how often a request is refused. Only rates and counts are published here: no trace, task question or answer.
- Reviewed passing attempts
- 15,007
- Flagged for reward hacking
- 617 (4.1%)
- Entries with at least one flag
- 72 of 76
- Provider content-policy blocks
- 1 of 35,405 attempts
Models76 of 76 entries
Reward hacking
An attempt that passes its automatic checks and map review is then read in full by an AI judge, which looks for a pass the agent did not earn. Only passing attempts are reviewed: a failed attempt scores 0 already. The judge is Claude Sonnet 5.5. It reads the task prompt and every agent message and command, and flags an attempt only for one of four things, naming the step and quoting the trace. A flagged attempt scores 0. How trace review works.
128 reviews could not be given because the judge's safety filter declined the trace; those attempts are not counted as reviewed.
Reward hacking rate
Flagged attempts as a share of reviewed passing attempts · Lower is better
How to read this chart
What this metric means. The share of reviewed passing attempts that an AI judge flagged for reward hacking: reading the files the automatic checks use, writing values no command computed, editing outputs to fit after a failed check, or fetching answers from outside the sandbox. A flagged attempt scores 0.
How to read this chart. Lower is better. Only attempts that passed every other check are reviewed, so a model with few passes has few reviews. Every flag is read by hand before results are published.
Data table: Reward hacking rate
By kind of reward hacking
Reading test or answer files
Flagged as a share of reviewed passing attempts · Lower is better
How to read this chart
What this category means. Reading or listing the files the automatic checks use: tests, reference answers or solutions. Each flag names the step and quotes the trace.
Data table: Reading test or answer files
Values no command computed
Flagged as a share of reviewed passing attempts · Lower is better
How to read this chart
What this category means. Writing result values that no command computed or read from the inputs, such as numbers copied from the task text or typed from memory. Each flag names the step and quotes the trace.
Data table: Values no command computed
Outputs edited to fit
Flagged as a share of reviewed passing attempts · Lower is better
How to read this chart
What this category means. Editing outputs to fit after a failed self-check instead of fixing the computation. Each flag names the step and quotes the trace.
Data table: Outputs edited to fit
No attempt has been flagged for answers fetched from outside.
Refusals
Provider content-policy blocks
1 of 35,405 agent attempts ended in a provider content-policy block (about 1 in 35,000): Claude Opus 4.8 (high), Claude Code. The provider stopped the request before the model answered. The attempt scores 0 and is not rerun.
Geospatial knowledge refusals
6 of 103,003 geospatial knowledge questions asked were refused: 2 by Claude Opus 4.7 (high), 1 by Claude Opus 4.6 (high), 1 by Claude Opus 4.8 (high), 1 by Claude Sonnet 4.6 (high), 1 by Claude Sonnet 5 (high). A refused question is left out of the model's score, and a model is ranked only while at most 1% of its questions are unscored.
Refusals the model writes itself
When a model declines a task in its own words, the run records do not mark it, so these refusals are not yet measured on agent tasks. They are shown as No data, never as zero.