EVE hallucination benchmark leaderboard
Marking the hallucinated passages in an answer about Earth observation, with no tools and no internet. A standalone evaluation: its scores are shown here and on model pages, but it is not counted in any index or ranking until more models have run it.
EVE hallucination score
Share of questions answered correctly, 0 to 100 · Higher is better
How to read this chart
Rows. Each model family once, at its best model and reasoning level, with at least 75% of the questions scored; the table lists every model and level, with how many questions each has scored.
Data table: EVE hallucination score
| Model | Score | Questions scored | Cost per question | Time per question (at 100 output tokens/s) |
|---|---|---|---|---|
| Claude Opus 5.5 (high) | 68 | 1,826 of 1,826 | $0.022 | 8s |
| Claude Sonnet 5.5 (xhigh) | 68 | 1,016 of 1,826 | $0.0096 | 7s |
| Claude Opus 5.5 (low) | 67 | 1,826 of 1,826 | $0.017 | 6s |
| Claude Sonnet 5.5 (high) | 67 | 1,826 of 1,826 | $0.0078 | 6s |
| Claude Opus 5 (high) | 67 | 1,522 of 1,826 | $0.028 | 14s |
| Claude Sonnet 5.5 (low) | 67 | 1,826 of 1,826 | $0.0083 | 5s |
| Claude Sonnet 5.5 (medium) | 67 | 1,826 of 1,826 | $0.0085 | 5s |
| Claude Fable 5.1 (high) | 66 | 494 of 1,826 | $0.053 | 9s |
| Nemotron 3.5 Super VL 120B (Reasoning) | 64 | 1,826 of 1,826 | $0.0016 | 50s |
| Step 5 Preview (Reasoning) | 64 | 1,826 of 1,826 | $0.010 | 308s |
| Claude Haiku 5.5 (xhigh) | 63 | 1,824 of 1,826 | $0.0014 | 11s |
| Claude Haiku 5.5 (high) | 63 | 1,824 of 1,826 | $0.00098 | 8s |
| GLM-5.3-Flash (low) | 62 | 1,826 of 1,826 | $0.00021 | 5s |
| Nemotron 3 Ultra (high) | 61 | 1,607 of 1,826 | $0.0041 | 162s |
| MiniMax M3.1 Flash (max) | 61 | 141 of 1,826 | $0.0014 | 533s |
| Nemotron 3.5 Lightning 30B (Reasoning) | 60 | 1,826 of 1,826 | $0.0012 | 72s |
| Claude Opus 5.5 (medium) | 51 | 38 of 1,826 | $0.021 | 8s |
| Claude Haiku 5.5 (low) | 50 | 191 of 1,826 | $0.00069 | 6s |
| Claude Opus 5.5 (xhigh) | 47 | 119 of 1,826 | $0.024 | 11s |
| Nemotron 3 Nano Omni 30B (Reasoning) | 46 | 1,826 of 1,826 | No data | 33s |
About this evaluation
Each item is a question about Earth observation and an answer that contains deliberately hallucinated passages. The model copies the answer and marks the passages that are false or unsupported, from its own knowledge, with no tools and no internet. 1,826 of the 1,830 items are scored; 4 have no reference passage.
The score is how far the marked passages overlap the reference ones, letter by letter, averaged over the items. Code checks it; no judge is used.
Published by ESA Φ-lab (EVE). Licence: CC BY 4.0.
