Introducing Geospatial Agent Benchmarks

Today we are publishing Geospatial Agent Benchmarks at axisspatial.com/arena. It runs AI models on GIS and Earth observation tasks (maps, spatial data and satellite imagery) in isolated sandboxes, checks every output with code against the source data, and ranks models on one headline score: the Geospatial Intelligence Index. It brings eight public benchmarks under one protocol.

Key findings

• The top is close. Claude Opus 5.5 (xhigh) leads both views of the index: 88.7 in Text only and 82.7 in Text + vision. GPT-6.1 Sol (xhigh) is 2.5 points behind in Text only, inside the noise, at about a third of the cost per task.
• Cheap models are not far behind. Claude Haiku 5.5 (xhigh) scores 82.0 at $0.017 per agent task, about a fifteenth of the leader's cost.
• The basics are saturated; the work is not. 14 of 21 ranked models score 90 or more on both guided workflows and geospatial knowledge. None reaches 90 on agent tasks or vision.
• Open-weights models trail by about 10 points. The best, DeepSeek V4 Pro (max), scores 79.0 and ranks 12th of 21.
• Every source benchmark had broken tasks. Audits excluded 119 tasks after running. Wrong answer keys alone had moved the Agent Index by up to 10.3 points, mostly against careful frontier models.

Research note. Results come from an ongoing evaluation, scored on the first attempt at each task. A model is ranked only when every part of the index is complete. Scores describe performance on these tasks under our setup, not in production. Axis Spatial builds geospatial AI products; none of its own models or agents is evaluated.

Why we built it

There was no independent answer to a simple question: which AI model is best for GIS and Earth observation work? Each geospatial benchmark released since 2025 covers one part of the work, with its own tools, agent, grader and models. Their scores cannot be compared or combined.

The tasks needed checking too. Every source we adopted had broken tasks. In one, the reference answer buffered fire stations by 2,500 degrees instead of 2,500 metres.

General leaderboards do not fill the gap. They measure coding, maths and general knowledge, not whether a model can join the right layers, compute a dryness index from satellite rasters, or decline a request the data cannot answer.

So we applied the method Artificial Analysis uses for general AI to geospatial work: one setup for every model, scores and cost side by side, every choice published and versioned.

What it measures

The Geospatial Intelligence Index asks which model is best at geospatial work overall. It is a weighted average of four groups, out of 8, with agent tasks counting most.

The index has two views. Text + vision uses all four groups. Text only leaves Vision out and weights the rest 4:1:1, so models without image input are ranked too. A model that cannot take images is never scored 0 for it. Text only still includes satellite data: there, models analyse rasters with code rather than looking at images.

The Geospatial Agent Index asks which agent set-up to use: a model, in a harness, at one reasoning level. It is the plain average of three evaluations, 488 tasks in all.

In the Text + vision view a fourth evaluation, Vision agents (tasks from ThinkGeo and OpenEarthAgent), adds work where the agent must look at images while it works. It is marked preliminary.

Each model runs in its developer's own harness where we run one: Claude Code for Claude models, Codex for OpenAI models, Kimi Code for Kimi models. Every other model runs in Terminus-2. The Intelligence Index shows one row per model, at its highest ranked reasoning level; the Agent Index compares harnesses.

First results

21 models are ranked in the Text only view and 12 in Text + vision. Anthropic and OpenAI models fill the top ten of both.

In Text + vision, Claude Opus 5.5 (xhigh) leads at 82.7, ahead of Claude Sonnet 5.5 (xhigh) at 78.9 and GPT-6.1 Sol (high) at 78.1. That 3.8-point gap is only just above the noise: it suggests a lead but does not prove one.

Cost. Cost per task varies far more than score. Cost is tokens at public list prices, with failed attempts included.

Reasoning level. Above high, extra reasoning adds little. On the Agent Index, Claude Sonnet 5.5 gains 7 points from medium to high, then 1 to xhigh. GPT-6.1 Sol scores 82 at both medium and xhigh.

Harness. Most same-model gaps between harnesses are small. Kimi K2.6 and K2.7 Code differ by about a point between Kimi Code and Terminus-2. The exception is small open models run in Codex: gpt-oss-20b (high) scores 33 in Terminus-2 and 9 in Codex.

Saturation. 14 of 21 ranked models score 90 or more on both guided workflows and geospatial knowledge. No ranked model reaches 90 on agent tasks or vision. Harder tasks, not new weights, are what will separate the leaders.

Open-weights models

Open-weights models, whose weights anyone can download and run, matter most to teams that need to keep data and models on their own infrastructure. On geospatial work they trail the proprietary leaders by about 10 points.

The best, DeepSeek V4 Pro (max), ranks 12th of 21 at 79.0. Kimi K3 and GLM-5.2 (max) sit just behind it. The gap looks wider than on general benchmarks: Artificial Analysis put the best open-weights models within 3 to 6 points of the proprietary leaders on its Intelligence Index in April 2026. The two indexes use different tasks, so the numbers are not directly comparable.

No open-weights model is ranked in the Text + vision view yet, because none has completed the vision parts; some take no image input at all.

We took three steps to make the comparison fair:

• Each model in a harness that suits it. Every open-weights model runs in Terminus-2, and some also in their developers' harnesses. We report harnesses separately in the Agent Index and never mix them in one row.
• Output tokens instead of time. Open-weights models were served mostly through one cloud provider, slower than the fastest hosts. Timing them would have measured the server, not the model, so the charts show output tokens per task. Every model gets the same 2-hour limit, and timed-out attempts are rerun.
• The same checks and the same judge. Open-weights and proprietary models are graded by the same code checks and the same map and trace judge.

We will report this gap with every release, so readers can see whether open-weights models are closing it on geospatial work.

How it is scored and checked

An attempt passes only if every required check passes. A plausible-looking answer is not enough.

1. Code checks against the source data. Values, coordinate reference systems, units, geometry, rasters and output files are checked by code, with stated tolerances. A GS-QA2 distance, for example, must be within 5% and a place name must match.
2. A map judge where a map is the output. Claude Sonnet 5.5 grades each map against a yes-or-no checklist. It passed 143 of 145 reference maps before use, and a check on 120 Claude and 120 other attempts showed no lean towards Claude models.
3. A trace review for reward hacking. The same judge reads every passing trace for shortcuts such as searching the disk for answer files or writing values that were never computed. A flagged attempt scores 0. Of 17,651 traces reviewed, 600 were flagged, and a model with too many unread flags is held until a person reads them.

Each agent attempt runs in a fresh sandbox with no network access; a separate sandbox runs the private checker. Scores are pass@1 on the first attempt at each task. Agent tasks have a 2-hour limit. A safety refusal scores 0 and is not rerun. Attempts cut short by our own infrastructure are never scored; they are rerun.

Auditing the tasks. Before any model ran, 167 tasks were left out because their references did not reproduce or could not be checked. After running, audits excluded 119 more that were broken rather than hard. They found wrong answer keys, "reject" tasks the data could answer, and one GS-QA2 key that counted whole rivers rather than the part inside the county, so careful models failed where a weaker one passed. Each exclusion applies to every model and is listed publicly with its reason.

The method is public and versioned at axisspatial.com/arena/methodology. Scoring never changes without a new version.

What this does not show

• Public tasks. Every task comes from a published benchmark, so models may have seen tasks or reference solutions in training.
• Our setup, not the original papers'. Scores are our results under our adaptation, not reproductions of numbers the benchmark authors published.
• One attempt per task. The full protocol is three attempts; we score the first for every model before running more. Read the ranking as observed performance, not a precise order.
• Noise. Redrawing the tasks 2,000 times puts differences under about 3 points within noise. That covers task choice only, so real noise is larger.
• Mixed harnesses. Part of a gap between two models can come from the harness rather than the model.
• One AI judge. Maps and traces are graded by one judge, tested and spot-checked by hand but not infallible.
• One provider per model. The same model served elsewhere may behave and cost differently.
• Coverage. Some models, including Google's Gemini entries, are not yet ranked because parts of the index are incomplete. Vision agents is preliminary.
• Scope. The benchmark does not test interactive desktop GIS or production deployments.

What comes next

Geospatial Agent Benchmarks is a non-commercial research project. No model developer pays to be listed, and every model runs on the same tasks with the same checks.

We built and published it in nine days, from 8 to 16 October 2026, through 33 methodology versions. The next releases will:

• finish the models not yet ranked, so every model in the list has a complete index;
• add harder agent tasks, starting with GeoAgentEval and GeoNatureAgent, because the leaders no longer separate on the current ones;
• retire or replace the guided workflows and geospatial knowledge groups if they stay saturated at the next version;
• add more vision work, starting with BigEarthNet.txt, and reasoning questions to geospatial knowledge;
• publish a release paper describing the method and results.

We will present the work at the AAI4EO workshop in Berlin on 20 October 2026.

If you build geospatial benchmarks, run spatial teams or develop models, tell us what you need. Suggest a model, report a task you think is broken, or offer tasks from your own work: get in touch.

Cite this work: Axis Spatial (2026). Geospatial Agent Benchmarks. https://www.axisspatial.com/arena

Sources

• Axis Spatial, Geospatial Agent Benchmarks and methodology. Figures in this note are from the scoring run of 16 October 2026, methodology 0.33; the live pages update as runs complete.
• Source benchmarks, adapted by Axis Spatial, with thanks to their authors:
  • Earth-Bench V2, OpenDataLab (Earth-Agent): Hugging Face. CC BY-NC 4.0; used with attribution, and only scores are published.
  • GeoBenchX, Krechetova and Kochedykov, 2025: GitHub
  • GS-QA2, Shang et al., UC Riverside: Hugging Face
  • GeoAgentBench, GeoX-Lab: GitHub
  • GeoAnalystBench, GeoDS Lab: GitHub
  • ThinkGeo, MBZUAI: GitHub; OpenEarthAgent, MBZUAI: GitHub
  • EVE, ESA Φ-lab: Hugging Face
• Artificial Analysis, Intelligence benchmarking methodology, whose method this work follows, and Recent open weights model launches, 30 April 2026, for the general open-weights gap.