If your team is choosing an AI model for GIS or Earth observation work, you can now see how each one does on that work, and what it costs, before you commit.
Today we are publishing the Geospatial Intelligence Index at axisspatial.com/
Key findings
- The top is close; the cost is not. Claude Opus 5.5 (xhigh) leads both views of the index: 88.7 in Text only and 82.7 in Text + vision. GPT-6.1 Sol (medium) is 2.3 points behind in Text only, inside the noise, at about a fifth of the cost per task.
- Long tasks are where agents fail. The leader passes 97% of short agent tasks and 70% of long ones. Every model drops the same way.
- Seeing is the weakest skill, and tools do not fix it. The best score on questions about satellite and aerial images is 59 out of 100. Giving the same models a full agent with tools moved their scores by between −6 and +7 points.
- The open-weights gap is mostly about knowing when to say no. Kimi K3, among the best open-weights models, is 9 points behind the leader on raster and remote sensing work but 32 points behind on requests the data cannot answer.
- A correct output is not proof of correct work. Reviewing the traces of passing attempts found agents taking shortcuts, and the rate depends on the harness: the same DeepSeek model had 2% of its passes flagged in Terminus-2 and 20% in Codex.
- Every source benchmark had broken tasks. Audits excluded 119 tasks after running, each listed with its reason. Wrong answer keys hurt careful models the most.
How general AI answers "which model is best"
General AI is evaluated in two layers, and geospatial work has only the first.
| Layer | Examples | What it gives you |
|---|---|---|
| Single benchmarks | SWE-bench, Terminal-Bench, GPQA, Humanity's Last Exam | One kind of work, checked one way: a repository's tests, a task's test script, a fixed answer |
| Aggregators | Artificial Analysis, Epoch AI, Stanford HELM, LMArena | Many benchmarks run again under one setup, so every model is compared on the same terms, often with cost beside the score |
A team choosing a model for coding reads the second layer, not twenty papers. The aggregator decides how every model is run and how every answer is checked, and publishes those choices.
Geospatial work has the first layer, published across papers since 2025, but not the second. General leaderboards do not fill the gap: they measure coding, maths and general knowledge, not whether a model can join the right layers, compute a dryness index from satellite rasters, or decline a request the data cannot answer. So there was no independent answer to a simple question: which AI model is best for GIS and Earth observation work?
Geospatial benchmarks are fragmented three ways
The geospatial benchmarks differ in the work they test, the agent that runs them and how they decide a pass. A score from one cannot be compared with a score from another, or combined with it.
| Benchmark | What it tests | Data | Agent its authors used | How a pass is decided |
|---|---|---|---|---|
| Earth-Bench | Computing over Earth observation products | Spectral and product rasters, RGB images | Earth-Agent, with its own tool set | Final answer plus tool-use steps |
| GeoBenchX | Multi-step analysis, maps, impossible requests | Country and city vectors, rasters, tables | Tool-calling agent with 23 functions | LLM judge comparing tool calls with a reference |
| GS-QA2 | Spatial questions over a map database | OpenStreetMap and elevation in PostGIS | Text-to-SQL and retrieval methods | Reference query with tolerances |
| ThinkGeo | Perception tools on aerial images | Optical imagery | ReAct agent with 14 tools | Step metrics and an LLM judge |
| OpenEarthAgent | Tool use over imagery and GIS data | RGB, SAR, multispectral, GIS layers | Its own tool registry and a fine-tuned 4B model | Step metrics and an LLM judge |
| GeoAgentBench | Multi-step GIS workflows | Vector, raster, NetCDF | Plan-and-React agent | Parameter accuracy and a map judge |
| GeoAnalystBench | GIS workflow and code generation | Vector, raster, tables | Direct generation, no agent | Code similarity and workflow validity |
The work. Geospatial work is not one kind of task. It spans vector analysis, raster and remote sensing computation, queries over spatial databases, map making and reading imagery, each with its own data, tools and outputs: a number, a file, a map, or a decision not to answer. Each benchmark tests one slice. None covers all of it.
The agent. Each benchmark drives models through its authors' own research harness, usually a small ReAct or tool-calling loop. An agent is a model in a harness, so every paper tests a different agent. That measures a model in a set-up nobody uses at work, and a different set-up in every paper.
The grading. This is the least visible difference and the one that matters most. One benchmark checks the final answer. One asks an LLM whether the tool calls look like a reference. One measures how similar generated code is to a reference script. Each decides a pass with a single check, and every kind of check has a blind spot: comparing tool calls passes an agent that called the right tools and wrote the wrong file; checking only the output cannot tell an answer computed from one found on the disk. A 70% in one paper and a 70% in another do not mean the same thing.
Two more problems sit under these. The tasks themselves needed checking: every source we adopted had broken tasks, with wrong answer keys or "impossible" requests the data could in fact answer. And images matter more here than in most fields, yet are tested least. Earth observation is imagery, and published agent studies find perception the weakest step: the OpenEarthAgent paper reports GPT-4o at 44 F1 on perception steps against 96 on GIS tool steps, and the ThinkGeo paper reports no model getting more than about 12% of final answers right. Other studies find text-only models do surprisingly well on some remote sensing question sets, so the value of seeing has to be measured, not assumed.
What we built: one protocol for eight benchmarks
We applied the method Artificial Analysis uses for general AI to geospatial work: one setup for every model, score and cost side by side, every choice published and versioned. This first version does not write new tasks. It takes the public benchmarks that already exist, audits them, and runs them all the same way.
The same harnesses. We run each model the way people actually use it: inside the agent harness its developer ships. Claude models run in Claude Code, OpenAI models in Codex, and Kimi models in Kimi Code once their runs there are complete (Kimi K3 is ranked in Terminus-2 for now). Every other model runs in Terminus-2, the open harness from the Terminal-Bench team. Every run goes through Harbor, their evaluation framework. Each entry is named harness first, as Artificial Analysis does for coding agents: "Claude Code – Claude Opus 5.5 (xhigh)".
The same sandbox. Each attempt runs in a fresh sandbox with no network access, with its data handed over in a folder. In most tasks the agent must hand back a value, a file or a map. A separate sandbox runs the private checker, so the agent never sees the answer key.
The same grading. Every attempt from every model goes through the same layers of checks, described in the next section, and scores pass@1 on the first attempt.
Two indexes. The Geospatial Intelligence Index asks which model is best at geospatial work overall. It is a weighted average of four groups, out of 8, with agent tasks counting most.
| Group | Weight | Items | What the model does |
|---|---|---|---|
| Agent tasks | 4 | 488 | Completes GIS and Earth observation tasks in a sandbox: the Geospatial Agent Index |
| Vision | 2 | 136 | Answers questions about satellite and aerial images, one call each, no tools |
| Guided workflows | 1 | 69 | Follows given steps to produce GIS outputs, from GeoAgentBench and GeoAnalystBench |
| Geospatial knowledge | 1 | 1,685 | Answers closed-book questions from ESA Φ-lab's EVE set |
The index has two views. Text + vision uses all four groups. Text only leaves Vision out and weights the rest 4:1:1, so models without image input are ranked too, and are never scored 0 for it. Text only still includes satellite data: there, models analyse rasters with code rather than looking at images.
The Geospatial Agent Index asks which agent set-up to use: a model, in a harness, at one reasoning level. It is the plain average of Earth-Bench V2 (117 tasks, such as a dryness index from vegetation and temperature rasters), GeoBenchX (150, multi-step spatial work with numbers, lists and maps, including requests to reject) and GS-QA2 (221, questions about US places and terrain from OpenStreetMap and elevation data in PostGIS). In the Text + vision view a fourth evaluation, Vision agents (64 tasks from ThinkGeo and OpenEarthAgent), adds tasks where the agent must look at images while it works. It is marked preliminary.
How a pass is decided
A pass means the same thing for every task here, whichever benchmark it came from: the attempt got through every layer.
- 1 · Audit the taskChecks: the reference reproduces, the key matches the question, the data can answer itCaught: 167 tasks left out before running and 119 after, such as a key that measured a whole river
- 2 · Check the output with codeChecks: values, CRS, units, geometry and files against the answer key and source data, with tolerancesReplaces: GeoBenchX's LLM comparison of tool calls. A refusal is checked both ways
- 3 · Judge where code cannot decideChecks: each map against a yes-or-no checklist, by one calibrated judge model for every entryCaught: the first judge failed calibration; the current one passed 143 of 145 reference maps
- 4 · Review every passing traceChecks: the agent's full trace for shortcuts, by a judge model, with a person reading every flagCaught: the same DeepSeek model flagged on 2% of passes in Terminus-2 and 20% in Codex
Pass: counts towards the score
1. Audit the task. Before any model ran, 167 tasks were left out because their references did not reproduce or could not be checked. After running, audits excluded 119 more that were broken rather than hard: wrong answer keys, "reject" tasks the data could answer, and GS-QA2 keys that counted whole rivers and canals rather than the part inside a county. There, a careful agent that clipped the river to the county failed, and a weaker one passed. Each exclusion applies to every model and is listed publicly with its reason.
2. Check the output with code. Code checks what the agent hands back against the answer key and the source data: values, coordinate reference systems, units, geometry, rasters and files, with stated tolerances. A GS-QA2 distance must fall within the source's tolerance and the place name must match. For GeoBenchX we replaced the original LLM comparison of tool calls with code checks of the outputs. About one in eight agent tasks is impossible by design: answering one fails, and so does refusing a task the data can answer.
3. Judge where code cannot decide. Whether a map shows the right thing in the right places is not something code can check. A judge model, Claude Sonnet 5.5, grades each map against a yes-or-no checklist written for that task. We treat the judge as part of the instrument: it had to grade the reference maps correctly before use. The first judge we tried failed, reading the palest colours and tiny islands as missing data. The current one passed 143 of 145 and was tested for any lean towards Claude models. The same judge grades every model, and changing it means re-judging everything.
4. Review every passing trace. An output can be right because the agent found the answer rather than worked it out. So a judge model reads the full trace of every passing attempt, looking for shortcuts such as searching the disk for answer files, writing values that were never computed, or fitting the output to the check. A flagged attempt scores 0, and a person reads every flag before results are published. Reading the flags by hand also corrected the reviewer: an agent listing its own folders is not cheating; an agent searching the whole disk for answers is. An attempt still waiting for review counts as a pass until it is reviewed.
The rules around it. Scores are pass@1 on the first attempt at each task, with a 2-hour limit per agent task. A safety refusal scores 0 and is not rerun. Attempts cut short by our own infrastructure are never scored; they are rerun. The all-pass rule follows Harvey's long-horizon legal agent benchmark. The method is public and versioned at axisspatial.com/arena/methodology, and scoring never changes without a new version.
Who it is for
One overall rank is the least useful thing the benchmark offers. Because every task runs under the same protocol, results can be split by the kind of work, so each reader can test on the work they actually do.
| Reader | Question | Where to look |
|---|---|---|
| Teams building geospatial agents | How does my model and harness compare with Claude Code, Codex and Terminus-2, and where does it break? | Agent Index by harness; splits by kind of work, data type, task length and task format |
| Organisations choosing a set-up | Which model, harness and reasoning level fits our work and budget? | Score beside cost per task, for raster and remote sensing, vector and overlay, mapping, or infeasible requests |
| Researchers | How does a new agent or method compare, on one kind of task, under one protocol? | Eight benchmarks with one grading method, every exclusion listed with its reason |
First results
Of 61 models and reasoning levels, 44 are ranked in the Text only view and 20 in Text + vision. Anthropic and OpenAI models fill the top ten of both.
- Claude Opus 5.5 (xhigh)88.7
- GPT-6.1 Sol (xhigh)86.2
- GPT-6 Sol (high)84.9
- Claude Opus 5 (high)84.5
- Claude Sonnet 5.5 (xhigh)84.4
- Claude Opus 4.8 (high)82.5
- Claude Haiku 5.5 (xhigh)82.0
- GPT-5.6 Sol (high)81.3
- Claude Sonnet 5 (high)80.9
- Claude Opus 4.7 (high)80.8
In Text + vision, Claude Opus 5.5 (xhigh) leads at 82.7, ahead of Claude Sonnet 5.5 (xhigh) at 78.9 and GPT-6.1 Sol (high) at 78.1. That 3.8-point gap is only just above the noise: it suggests a lead but does not prove one.
Cost varies far more than score. Cost is tokens at public list prices, with failed attempts included.
| Entry | Text only index | Cost per agent task |
|---|---|---|
| Claude Code – Claude Opus 5.5 (xhigh) | 88.7 | $0.25 |
| Codex – GPT-6.1 Sol (medium) | 86.4 | $0.046 |
| Codex – GPT-6 Luna (xhigh) | 80.0 | $0.0070 |
Reasoning level. Above high, extra reasoning mostly buys cost, not score. GPT-6.1 Sol's medium, high and xhigh levels span 0.7 points while cost rises from $0.046 to $0.085. Claude Sonnet 5.5 gains 5.4 points from medium to high, then 0.1 to xhigh.
Harness. Most same-model gaps between harnesses are within the noise. Kimi K2.6 and K2.7 Code differ by under a point between Kimi Code and Terminus-2. The exception is open models run in Codex: gpt-oss-20b loses 24.17 points against Terminus-2. The harness also changes behaviour, which only the trace review shows: DeepSeek V4 Flash (high) had 2% of its passing attempts flagged for shortcuts in Terminus-2 and 20% in Codex, and DeepSeek V4 Pro (high) 1% against 13%. Both Codex runs are still incomplete.
Saturation. 33 of 44 ranked entries score 90 or more on each of guided workflows and geospatial knowledge. No ranked entry reaches 90 on agent tasks or vision. Those two basics groups now meet our retirement threshold for the first time; harder tasks, not new weights, are what will separate the leaders.
Where agents fail
The 488 agent tasks can be split by kind of work, data type, task length and format, because they were all run and graded the same way. The splits show where the difficulty lies.
| Entry | Raster and remote sensing (181) | Vector and overlay (118) | Mapping (57) | Infeasible requests (60) | Short tasks (182) | Long tasks (165) |
|---|---|---|---|---|---|---|
| Claude Code – Claude Opus 5.5 (xhigh) | 89 | 94 | 88 | 82 | 97 | 70 |
| Codex – GPT-6.1 Sol (medium) | 83 | 94 | 81 | 90 | 97 | 69 |
| Terminus-2 – Kimi K3 (Reasoning) | 80 | 76 | 81 | 50 | 90 | 58 |
Long tasks cost every model about 30 points. Optical multispectral imagery is the hardest data type even when models work on it through code: the leader scores 74 there, against 94 on elevation and 90 on vector-only tasks.
Seeing is the weakest skill. The same 136 image questions were answered twice: in one call with no tools, and by the model inside its agent harness, where it can crop, zoom and compute on the image.
One call, no tools In its agent harness, with tools
- Claude Haiku 5.5 (medium)45 → 52 (+7)
- Claude Opus 5.5 (medium)57 → 63 (+6)
- GPT-6.1 Sol (medium)55 → 58 (+3)
- Claude Sonnet 5.5 (medium)56 → 55 (−1)
- GPT-6 Luna (xhigh)50 → 44 (−6)
Tools helped some models a little and hurt others; the ceiling stays near 60. Small open-weights models hold up on images: Gemma 4 26B A4B scores 64 on the Earth-Bench V2 image questions, within 2 points of Claude Sonnet 5 (66), and Qwen 3.8 27B, still being scored, is level with GPT-6.1 Sol (high) at 53 overall.
Open-weights models
Open-weights models, whose weights anyone can download and run, matter most to teams that need to keep data and models on their own infrastructure. On geospatial work they trail the proprietary leaders by about 10 points.
- Claude Opus 5.5 (xhigh), for reference88.7
- DeepSeek V4 Pro (max)79.0
- Kimi K3 (Reasoning)78.6
- GLM-5.2 (max)78.4
- DeepSeek V4 Flash (max)77.2
- GLM-5.3 (max)77.1
- GLM-5.3-Flash (max)71.0
- Gemma 4 26B A4B (Reasoning)64.4
- GLM-4.7-Flash (Reasoning)49.6
- Mistral Large 3 (Non-reasoning)44.5
- gpt-oss-120b (low)44.3
The best, DeepSeek V4 Pro (max), ranks 22nd of 44 at 79.0, counting every reasoning level. GLM-5.2 (high) and Kimi K3 tie just behind it. The gap is wider than on general benchmarks: Artificial Analysis put the best open-weights models within 3 to 6 points of the proprietary leaders on its Intelligence Index in April 2026. The two indexes use different tasks, so the numbers are not directly comparable, but geospatial agent work looks harder to catch up on.
The gap is not spread evenly. On raster and remote sensing work and on mapping, the best open-weights models are within about 10 points of the leaders. On recognising requests the data cannot answer they are about 30 behind. That gap would not show in a benchmark that only checks answers to solvable questions.
No open-weights model is ranked in the Text + vision view yet, because none has completed the vision parts; some take no image input at all.
We took three steps to make the comparison fair:
- Each model in a harness that suits it. Every open-weights model runs in Terminus-2, and some also in their developers' harnesses. Harnesses are reported separately in the Agent Index and never mixed in one row.
- Time at a common speed. Serving speed differs by provider, so time is shown at a common 100 output tokens per second. Every model gets the same 2-hour limit, and timed-out attempts are rerun.
- The same checks and the same judges. Open-weights and proprietary models go through the same four layers, with the same judge model for maps and traces.
We will report this gap with every release, so readers can see whether open-weights models are closing it on geospatial work.
What this does not show
- Public tasks. Every task comes from a published benchmark, so models may have seen tasks or reference solutions in training.
- Our setup, not the original papers'. Scores are our results under our adaptation and our grading, not reproductions of numbers the benchmark authors published.
- One attempt per task. The full protocol is three attempts; we score the first for every model before running more. Read the ranking as observed performance, not a precise order.
- Noise. Redrawing the tasks 2,000 times gives a gap between two entries a 95% interval of about ±3.3 points. That covers task choice only, so real noise is larger.
- Mixed harnesses. Part of a gap between two models can come from the harness rather than the model.
- One judge model. Maps and traces are graded by one judge, calibrated and spot-checked by hand but not infallible.
- One provider per model. The same model served elsewhere may behave and cost differently.
- Coverage. Some models, including Google's Gemini entries, are not yet ranked because parts of the index are incomplete. Vision agents is preliminary.
- Scope. The benchmark does not test interactive desktop GIS or production deployments.
What geospatial agent evaluation still needs
Putting the public benchmarks under one protocol also shows what none of them has yet. Above all, length.
| Per task | Geospatial agent tasks today (leader) | Coding agents on Artificial Analysis (longest) |
|---|---|---|
| Turns or time | About 7 turns | Up to 90 minutes |
| Output or total tokens | About 5,000 output tokens | Up to 28 million tokens |
| Cost | $0.25 | Up to $14 |
Real geospatial projects, such as a flood-exposure assessment or a land-cover change study, take hours and dozens of steps. No public geospatial benchmark tests work of that length, and our results already show agents failing as tasks get longer.
- Finding the data. Every task today hands the agent its data in a folder. Real work starts with searching catalogues and cloud archives. Benchmarks that need live services, such as Earth Engine, cannot yet run in a sealed, repeatable sandbox.
- Matched tests of vision. The same task with and without the agent being able to see the images, so the value of vision can be measured rather than argued.
- More kinds of data. SAR, hyperspectral, LiDAR and point clouds are barely covered; 49 of ThinkGeo's 50 SAR images are missing from its release.
- Professional software. Nothing tests an agent working in desktop GIS such as QGIS or ArcGIS.
- Work products graded like a senior analyst would. Deliverables reviewed against expert rubrics, not only a single value.
- Private, held-out tasks. Public tasks can leak into training data; a held-out set is the only check.
- Clear licences and released data. Most of the other datasets we reviewed could not be used: unreleased, gated, non-commercial or unclear terms.
What comes next
The Geospatial Intelligence Index is a non-commercial research project. No model developer pays to be listed, and every model runs on the same tasks with the same checks.
The next releases will:
- finish the entries not yet ranked, so every model in the list has a complete index
- add harder and longer agent tasks, starting with GeoAgentEval and GeoNatureAgent
- run the matched vision test: the same tasks with and without images
- add more vision work, starting with BigEarthNet.txt, and reasoning questions to geospatial knowledge
- add benchmarks we reviewed but could not run yet, as their data is released or can run offline: TerraBench, OpenEarth-Bench and Earth-Bench-OW among them
- add held-out variants of public tasks on new areas, dates or thresholds, kept private
- retire or replace the guided workflows and geospatial knowledge groups if they meet the saturation rule again
- publish a release paper describing the method and results
If you have a geospatial dataset with long, real tasks, especially one that cannot be released publicly, we would like to evaluate on it. We will present the work at the AAI4EO workshop in Berlin on 20 October 2026.
If you build geospatial benchmarks, run spatial teams or develop models, tell us what you need. Audit our exclusions, report a task you think is broken, suggest a model or a harness to test, or offer tasks from your own work: Work with us.
Cite this work: Axis Spatial (2026). Geospatial Intelligence Index. https://www.axisspatial.com/arena
Sources
Axis Spatial, Geospatial Intelligence Index and methodology. Figures in this note are from Version 1.0, results as of 19 October 2026; the live pages update as runs complete.
Source benchmarks, adapted by Axis Spatial, with thanks to their authors:
- Earth-Bench V2, OpenDataLab (Earth-Agent): Hugging Face. CC BY-NC 4.0; used with attribution, and only scores are published.
- GeoBenchX, Krechetova and Kochedykov, 2025: GitHub
- GS-QA2, Shang et al., UC Riverside: Hugging Face
- GeoAgentBench, GeoX-Lab: GitHub
- GeoAnalystBench, GeoDS Lab: GitHub
- ThinkGeo and OpenEarthAgent, MBZUAI: ThinkGeo on GitHub, ThinkGeo on Hugging Face; OpenEarthAgent on GitHub, OpenEarthAgent on Hugging Face
- EVE, ESA Φ-lab: Hugging Face
General AI evaluation, for the two layers: SWE-bench, Terminal-Bench, GPQA and Humanity's Last Exam; Artificial Analysis, Epoch AI, Stanford HELM and LMArena.
Artificial Analysis, Intelligence benchmarking methodology, whose method this work follows, and Recent open weights model launches, 30 April 2026, for the general open-weights gap. Artificial Analysis, Claude Code vs Codex, for naming entries by harness and model, and for cost, time and tokens per coding-agent task (accessed 19 October 2026).
Harbor, the Terminal-Bench team's framework for evaluating agents such as Claude Code, Codex CLI and OpenHands.
Harvey, Open-Sourcing Harvey's Long Horizon Legal Agent Benchmark, for all-pass grading of agent work; unlike this benchmark, Harvey's tasks were written from scratch by legal experts.
Source benchmark papers, for the comparison table and the vision evidence: Earth-Agent and Earth-Bench, GeoBenchX, ThinkGeo, OpenEarthAgent, GeoAgentBench, GeoAnalystBench; and RSHR-Bench on text-only models doing well on some remote sensing question sets.
