Vision agents
Agents that look at aerial images and use code to find, count and measure what the question asks.
Preliminary, one attempt, not part of any index.
Ongoing evaluation: results update as runs complete. Last updated 14 October 2026.
Scores and cost
Vision agents score
Share of tasks passed, every named answer right, 0 to 100 · Higher is better
How to read this chart
Agent and model. Each bar is a model in a particular agent harness. The harnesses differ (their tools, prompts and limits are their own), so a difference between bars is not a difference between models alone.
What this metric means. The average pass rate over the tasks decided, each task weighted equally. A task passes only if every named answer is right.
Data table: Vision agents score
| Agent and model | Vision agents score | Tasks decided |
|---|---|---|
| Claude Code – Claude Opus 5.5 (medium) | 56.6 | 106 of 106 |
| Codex – GPT-6.1 Sol (medium) | 56.6 | 106 of 106 |
| Claude Code – Claude Sonnet 5.5 (medium) | 46.2 | 106 of 106 |
| Stirrup – GLM-5.3-Flash (low) | 43.4 | 106 of 106 |
| Claude Code – Claude Haiku 5.5 (medium) | 42.5 | 106 of 106 |
| Codex – GPT-6 Luna (xhigh) | 42.5 | 106 of 106 |
| Stirrup – Kimi K2.7 Code (always on) | 36.8 | 106 of 106 |
| Stirrup – Kimi K2.6 (always on) | 35.8 | 106 of 106 |
| Stirrup – Gemma 4 26B A4B (thinking on) | 19.0 | 105 of 106 |
| Stirrup – GLM-5.3-Flash (max) | 17.0 | 106 of 106 |
ThinkGeo: vision agents score
Share of tasks passed, 0 to 100 · Higher is better
How to read this chart
Sources. ThinkGeo: 81 tasks; OpenEarthAgent: 25 tasks. Same scale and order in both charts, so an entry's two bars compare directly.
Data table: ThinkGeo: vision agents score
| Agent and model | ThinkGeo score |
|---|---|
| Claude Code – Claude Opus 5.5 (medium) | 51 |
| Codex – GPT-6.1 Sol (medium) | 52 |
| Claude Code – Claude Sonnet 5.5 (medium) | 42 |
| Stirrup – GLM-5.3-Flash (low) | 36 |
| Claude Code – Claude Haiku 5.5 (medium) | 37 |
| Codex – GPT-6 Luna (xhigh) | 36 |
| Stirrup – Kimi K2.7 Code (always on) | 33 |
| Stirrup – Kimi K2.6 (always on) | 30 |
| Stirrup – Gemma 4 26B A4B (thinking on) | 15 |
| Stirrup – GLM-5.3-Flash (max) | 15 |
OpenEarthAgent: vision agents score
Share of tasks passed, 0 to 100 · Higher is better
How to read this chart
Sources. ThinkGeo: 81 tasks; OpenEarthAgent: 25 tasks. Same scale and order in both charts, so an entry's two bars compare directly.
Data table: OpenEarthAgent: vision agents score
| Agent and model | OpenEarthAgent score |
|---|---|
| Claude Code – Claude Opus 5.5 (medium) | 76 |
| Codex – GPT-6.1 Sol (medium) | 72 |
| Claude Code – Claude Sonnet 5.5 (medium) | 60 |
| Stirrup – GLM-5.3-Flash (low) | 68 |
| Claude Code – Claude Haiku 5.5 (medium) | 60 |
| Codex – GPT-6 Luna (xhigh) | 64 |
| Stirrup – Kimi K2.7 Code (always on) | 48 |
| Stirrup – Kimi K2.6 (always on) | 56 |
| Stirrup – Gemma 4 26B A4B (thinking on) | 32 |
| Stirrup – GLM-5.3-Flash (max) | 24 |
Vision agents score vs. cost per task
Vision agents score; cost per task (USD) · Higher and further left is better
- Most attractive quadrant
- Pareto line
How to read this chart
How to read this chart. Each point is one model in one agent harness. Further left means a lower average cost per task (log scale); higher means a higher vision agents score. Both axes fit the points shown, so the score axis does not start at zero. The shaded most attractive quadrant is the top-left quarter of the plot, split at the middle of each axis: a reading aid, not a fixed bar. The dotted Pareto line joins the points that no cheaper one beats on score. Only exact costs are plotted; the table lists every entry.
Cost per task. What one attempt at a task costs at the provider's published API prices, averaged over attempts, failed ones included.
Agent and model. Each bar is a model in a particular agent harness. The harnesses differ (their tools, prompts and limits are their own), so a difference between bars is not a difference between models alone.
Data table: Vision agents score vs. cost per task
| Agent and model | Vision agents score | Cost per task | On the Pareto line |
|---|---|---|---|
| Claude Code – Claude Opus 5.5 (medium) | 56.6 | $0.090 | No |
| Codex – GPT-6.1 Sol (medium) | 56.6 | $0.037 | Yes |
| Claude Code – Claude Sonnet 5.5 (medium) | 46.2 | $0.034 | Yes |
| Stirrup – GLM-5.3-Flash (low) | 43.4 | $0.023 | Yes |
| Claude Code – Claude Haiku 5.5 (medium) | 42.5 | $0.0050 | Yes |
| Codex – GPT-6 Luna (xhigh) | 42.5 | $0.0065 | No |
| Stirrup – Kimi K2.7 Code (always on) | 36.8 | $0.69 | No |
| Stirrup – Kimi K2.6 (always on) | 35.8 | $0.83 | No |
| Stirrup – Gemma 4 26B A4B (thinking on) | 19.0 | $0.067 | No |
| Stirrup – GLM-5.3-Flash (max) | 17.0 | $0.22 | No |
Vision agents includes: 106 tasks, 81 from ThinkGeo, 25 from OpenEarthAgent, each passed only if every named answer is right, checked by code. See the methodology for how we run each evaluation.
All results
| Agent and model | ThinkGeo (81 tasks) | OpenEarthAgent (25 tasks) | Vision agents score | Tasks decided | Cost per task | Time per task |
|---|---|---|---|---|---|---|
| Claude Code – Claude Opus 5.5 (medium) | 51 | 76 | 57 | 106 of 106 | $0.090 | 0.3 min |
| Codex – GPT-6.1 Sol (medium) | 52 | 72 | 57 | 106 of 106 | $0.037 | 0.8 min |
| Claude Code – Claude Sonnet 5.5 (medium) | 42 | 60 | 46 | 106 of 106 | $0.034 | 0.2 min |
| Stirrup – GLM-5.3-Flash (low) | 36 | 68 | 43 | 106 of 106 | $0.023 | 4.6 min |
| Claude Code – Claude Haiku 5.5 (medium) | 37 | 60 | 42 | 106 of 106 | $0.0050 | 0.4 min |
| Codex – GPT-6 Luna (xhigh) | 36 | 64 | 42 | 106 of 106 | $0.0065 | 2.3 min |
| Stirrup – Kimi K2.7 Code (always on) | 33 | 48 | 37 | 106 of 106 | $0.69 | 14.7 min |
| Stirrup – Kimi K2.6 (always on) | 30 | 56 | 36 | 106 of 106 | $0.83 | 20.0 min |
| Stirrup – Gemma 4 26B A4B (thinking on) | 15 | 32 | 19 | 105 of 106 | $0.067 | 16.6 min |
| Stirrup – GLM-5.3-Flash (max) | 15 | 24 | 17 | 106 of 106 | $0.22 | 26.3 min |
Sources: ThinkGeo (CC BY 4.0) and OpenEarthAgent (Apache 2.0); scores only, never questions or images. Licences in the methodology.
Look and work: the vision questions with an agent
Look and work score
Share of vision questions answered correctly, 0 to 100 · Higher is better
How to read this chart
Agent and model. Each bar is a model in a particular agent harness, so a difference between bars is not a difference between models alone.
What this metric means. The plain average of the source scores: the share of questions answered correctly from each source.
Data table: Look and work score
| Agent and model | Earth-Bench V2 images (50 questions) | ThinkGeo (86 questions) | Score | Questions decided | Not supported by the route | Cost per question | Time per question |
|---|---|---|---|---|---|---|---|
| Claude Code – Claude Opus 5.5 (medium) | 74 | 51 | 63 | 136 of 136 | 0 | $0.087 | 0.3 min |
| Codex – GPT-6.1 Sol (medium) | 72 | 43 | 58 | 136 of 136 | 0 | $0.031 | 0.7 min |
| Claude Code – Claude Sonnet 5.5 (medium) | 66 | 43 | 55 | 136 of 136 | 0 | $0.036 | 0.2 min |
| Claude Code – Claude Haiku 5.5 (medium) | 66 | 37 | 52 | 136 of 136 | 0 | $0.0044 | 0.3 min |
| Codex – GPT-6 Luna (xhigh) | 56 | 33 | 44 | 136 of 136 | 0 | $0.0050 | 1.8 min |
The same 136 questions as Vision, but an agent views the image files and may run Python before answering; one attempt, not part of any index. See the methodology.