Mixed track
An agent looks at aerial images and works with code to reach several checked answers: it finds the objects the question names, measures them (distances, areas, counts), converts with the stated ground resolution and finishes the arithmetic. A task passes only if every named answer is right.
Preliminary. One attempt per task, not part of any index. Each row is a model in a particular agent harness, and the harnesses differ (their tools, prompts and limits are their own), so a difference between rows is not a difference between models alone.
Ongoing evaluation: results update as runs complete. Last updated 13 October 2026.
Results
| Agent and model | ThinkGeo (81 tasks) | OpenEarthAgent (25 tasks) | Mixed track score | Tasks decided | Cost per task | Time per task |
|---|---|---|---|---|---|---|
| Claude Code – Claude Opus 5.5 (medium) | 51 | 76 | 57 | 106 of 106 | $0.090 | 0.3 min |
| Codex – GPT-6.1 Sol (medium) | 52 | 72 | 57 | 106 of 106 | $0.037 | 0.8 min |
| Claude Code – Claude Sonnet 5.5 (medium) | 42 | 60 | 46 | 106 of 106 | $0.034 | 0.2 min |
| Stirrup – GLM-5.3-Flash (low) | 36 | 68 | 43 | 106 of 106 | $0.023 | 4.6 min |
| Claude Code – Claude Haiku 5.5 (medium) | 37 | 60 | 42 | 106 of 106 | $0.0050 | 0.4 min |
| Codex – GPT-6 Luna (xhigh) | 36 | 64 | 42 | 106 of 106 | $0.0065 | 2.3 min |
| Stirrup – Kimi K2.7 Code (always on) | 33 | 48 | 37 | 106 of 106 | $0.685 | 14.7 min |
| Stirrup – Kimi K2.6 (always on) | 30 | 56 | 36 | 106 of 106 | $0.833 | 20.0 min |
| Stirrup – Gemma 4 26B A4B (thinking on) | 15 | 32 | 19 | 105 of 106 | $0.067 | 16.6 min |
| Stirrup – GLM-5.3-Flash (max) | 15 | 24 | 17 | 106 of 106 | $0.220 | 26.3 min |
How to read these results
- 106 tasks: 81 from ThinkGeo, 25 from OpenEarthAgent. Each asks for one or more named answers (numbers within a stated tolerance, whole-number counts, labels or yes/no), 142 checks in all, checked by code; no AI judge decides an answer.
- The mixed track score is the average pass rate over the tasks decided, each task weighted equally; the source columns are the same over each source's tasks.
- 12 passes are provisional: their traces have not yet been reviewed for reward hacking, as in the main index.
- Every task is image measurement in several steps (detect, measure, convert, compare). None combines imagery with GIS data files, so this track does not yet test image-plus-GIS workflows.
- Sources: ThinkGeo (dataset MBZUAI/ThinkGeo at revision ae673e6, CC BY 4.0 as its dataset card declares; code mbzuai-oryx/ThinkGeo, Apache-2.0) and OpenEarthAgent (dataset MBZUAI/OpenEarthAgent at revision f4d4e18, Apache-2.0 as its dataset card declares). The images come from the remote-sensing datasets these draw on; for most of them the source and licence are not verified, so this site shows no images. This non-commercial site publishes scores only, never questions or images.
- Related: Geospatial image understanding (questions answered from images) and the methodology.