Axis Spatial Spatial Intelligence Index

Search modelsMenu

EVE hallucination benchmark leaderboard

Marking the hallucinated passages in an answer about Earth observation, with no tools and no internet. A standalone evaluation: its scores are shown here and on model pages, but it is not counted in any index or ranking until more models have run it.

EVE hallucination score

Share of questions answered correctly, 0 to 100 · Higher is better

EVE hallucination scoreScores 0 to 100: Claude Opus 5.5 (high): 68; Nemotron 3.5 Super VL 120B (Reasoning): 64; Step 5 Preview (Reasoning): 64; Claude Haiku 5.5 (xhigh): 63; GLM-5.3-Flash (low): 62; Nemotron 3 Ultra (high): 61; Nemotron 3.5 Lightning 30B (Reasoning): 60; Nemotron 3 Nano Omni 30B (Reasoning): 460255075100Claude Opus 5.5 (high)68Nemotron 3.5 Super VL 120B (Reasoning)64Step 5 Preview (Reasoning)64Claude Haiku 5.5 (xhigh)63GLM-5.3-Flash (low)62Nemotron 3 Ultra (high)61Nemotron 3.5 Lightning 30B (Reasoning)60Nemotron 3 Nano Omni 30B (Reasoning)46
How to read this chart

Rows. Each model family once, at its best model and reasoning level, with at least 75% of the questions scored; the table lists every model and level, with how many questions each has scored.

Data table: EVE hallucination score
EVE hallucination score (share of questions answered correctly, 0 to 100)
ModelScoreQuestions scoredCost per questionTime per question (at 100 output tokens/s)
Claude Opus 5.5 (high)681,826 of 1,826$0.0228s
Claude Sonnet 5.5 (xhigh)681,016 of 1,826$0.00967s
Claude Opus 5.5 (low)671,826 of 1,826$0.0176s
Claude Sonnet 5.5 (high)671,826 of 1,826$0.00786s
Claude Opus 5 (high)671,522 of 1,826$0.02814s
Claude Sonnet 5.5 (low)671,826 of 1,826$0.00835s
Claude Sonnet 5.5 (medium)671,826 of 1,826$0.00855s
Claude Fable 5.1 (high)66494 of 1,826$0.0539s
Nemotron 3.5 Super VL 120B (Reasoning)641,826 of 1,826$0.001650s
Step 5 Preview (Reasoning)641,826 of 1,826$0.010308s
Claude Haiku 5.5 (xhigh)631,824 of 1,826$0.001411s
Claude Haiku 5.5 (high)631,824 of 1,826$0.000988s
GLM-5.3-Flash (low)621,826 of 1,826$0.000215s
Nemotron 3 Ultra (high)611,607 of 1,826$0.0041162s
MiniMax M3.1 Flash (max)61141 of 1,826$0.0014533s
Nemotron 3.5 Lightning 30B (Reasoning)601,826 of 1,826$0.001272s
Claude Opus 5.5 (medium)5138 of 1,826$0.0218s
Claude Haiku 5.5 (low)50191 of 1,826$0.000696s
Claude Opus 5.5 (xhigh)47119 of 1,826$0.02411s
Nemotron 3 Nano Omni 30B (Reasoning)461,826 of 1,826No data33s

About this evaluation

Each item is a question about Earth observation and an answer that contains deliberately hallucinated passages. The model copies the answer and marks the passages that are false or unsupported, from its own knowledge, with no tools and no internet. 1,826 of the 1,830 items are scored; 4 have no reference passage.

The score is how far the marked passages overlap the reference ones, letter by letter, averaged over the items. Code checks it; no judge is used.

Published by ESA Φ-lab (EVE). Licence: CC BY 4.0.

Related links