Comparison view
Which benchmark
proves what?
Each row links to a full reference page: what the benchmark evaluates, how it verifies answers, what it can show, and what it cannot. Claims marked pending verification could not be confirmed from public sources.
All benchmarks
19 entries · click a column head to sortIn shortTwo columns do most of the work. Executes code? — only these benchmarks can show that an agent's program actually runs. Deterministic verify? — only these check answers by recomputation rather than by another model's opinion. Open any benchmark for the full evidence-boundary record.
| Benchmark | Tasks | Executes code? | Deterministic verify? | Open? | Data licence |
|---|---|---|---|---|---|
| GeoAnalystBench / GISclaw | 50 | yes | yes | yes | — |
| GeoBenchX | 202 | yes | no | partial | — |
| GEO-Bench-2 | 19 | no | yes | partial | varies by dataset |
| GEO-Bench-2 Leaderboard | — | no | yes | partial | — |
| GEOBench-VLM | pending | no | yes | partial | — |
| GeoMMBench / GeoMMAgent | 1053 | no | yes | yes | CC BY 4.0 |
| GABench / GeoAgentBench | 53 | yes | no | yes | — |
| MapQA | 3154 | no | yes | yes | pending verification |
| ThinkGeo | pending | pending verification | yes | no | — |
| Earth-Bench / Earth-Agent | pending | pending verification | yes | no | — |
| OpenEarth-Bench / OpenEarthAgent | pending | pending verification | yes | no | — |
| TerraBench | pending | pending verification | pending verification | no | — |
| RSRCC | 126000 | no | yes | no | — |
| UrbanSARFloods | 8879 | no | yes | no | — |
| GeoAI Agency Primitives | pending | yes | yes | no | — |
| Axis Synthetic Migration Cases | 3 | yes | yes | no | Axis original synthetic |
| GeoNatureAgent Benchmark | 93 | yes | yes | yes | underlying indicator d… |
| EO-Gym | 9078 | yes | yes | yes | follows the licences o… |
| MultiGlobeQA | 46,060 | yes | yes | yes | source knowledge-graph… |