Version 2 of the Geospatial Agent Index, current since 13 November 2026. It is the plain average of Earth-Bench V2, GeoBenchX and GS-QA2, with Vision agents added in the Text + vision view, and it is the Agent tasks group of the Geospatial Intelligence Index. The rules shared by every evaluation, the limitations and the data sources are on the methodology page.
Cite this index
Pokkiri, N. (2026). Geospatial Agent Index v2: Methodology. Axis Spatial. https://www.axisspatial.com/arena/methodology/agent-index
BibTeX:
@techreport{pokkiri2026gai,
author = {Pokkiri, Najah},
title = {Geospatial Agent Index v2: Methodology},
institution = {Axis Spatial},
year = {2026},
version = {2.0},
url = {https://www.axisspatial.com/arena/methodology/agent-index},
}Views and evaluations
The plain average of its evaluations' scores, from 0 to 100, so an evaluation with many tasks does not outweigh one with few.
| View | Evaluations | Used for |
|---|---|---|
| Text only | Earth-Bench V2, GeoBenchX and GS-QA2 | The ranking, the breakdowns and the cost, time and token figures; Agent tasks in the Text only Intelligence Index |
| Text + vision | The same three and Vision agents | Agent tasks in the Text + vision Intelligence Index |
Earth-Bench V2
Included in Geospatial Agent Index v2 at 1/3 weighting (Text only) and 1/4 (Text + vision).
Questions answered by computing over satellite products: temperatures, vegetation and dryness indices, rainfall, water vapour and night lights. The source gives the agent ready-made tools; here the agent writes Python instead.
GeoBenchX
Included in Geospatial Agent Index v2 at 1/3 weighting (Text only) and 1/4 (Text + vision).
Maps, numbers and lists from country and city data. Some tasks cannot be done with the data provided, and the right answer is to decline. Every task asks for the same output files, so the instruction does not reveal which kind it is.
GS-QA2
Included in Geospatial Agent Index v2 at 1/3 weighting (Text only) and 1/4 (Text + vision).
Questions about places and terrain in the United States, such as the nearest park to a landmark, the direction to a museum or the slope at a point, answered by querying a PostGIS database of OpenStreetMap features and elevation data in the sandbox.
Sample and tolerances
We use the release's balanced sample of 245 questions: up to 5 from each of 49 question templates, drawn with a fixed seed (20261011). The source's tolerances apply, for example 5 m for a location, 10 m for an elevation and 5° for a direction, and the score averages within each template first, so each kind of question weighs the same.
Vision agents
Included in Geospatial Agent Index v2 at 1/4 weighting (Text + vision only).
Agents that look at aerial images and use code to find, count and measure what the question asks, converting with the stated ground resolution. It is the fourth evaluation of the Geospatial Agent Index's Text + vision view, and preliminary: every task is image measurement, none yet combining imagery with GIS data files.
How the tasks were built, and which runs count
106 tasks were built from ThinkGeo (81) and OpenEarthAgent (25), with answers checked by code and no AI judge. A review excluded 42, listed in excluded tasks, and corrected 12, so 64 are scored; on the 4 whose questions were clarified, only reruns count.
A run counts in the index only at the entry's own reasoning level in its own harness; other runs are shown on the Vision agents page only. That page also shows the 136 Vision questions run with an agent that may view the images and run Python, which counts in no index.
Other evaluations
Standalone evaluations (not part of Geospatial Agent Index v2). Guided workflows count in the Geospatial Intelligence Index instead, and the standalone evaluations on the Evaluations page count in no index.
How attempts are run
Harnesses
Terminus-2 runs unmodified: the model proposes shell commands, sees the output and continues. Each developer's own agent runs as a separate entry, and results from different harnesses are never averaged together. Details: agent harnesses and model selection.
Sandbox, sampling and limits
| Setting | Value |
|---|---|
| Sandbox | An isolated cloud sandbox with no internet access, with Python and open geospatial libraries |
| Attempts | One scored attempt per task |
| Turns | Up to 60 |
| Time limit | 2 hours of agent time per Geospatial Agent Index task for every model. An attempt over the time limit scores 0 |
| Reasoning | As the entry's level says, set explicitly where the model has a control, and passed back between turns |
| Temperature | 0.6 with reasoning on, unless the model's developer recommends otherwise. Claude models in Claude Code and GPT models in Codex run with their agent's defaults |
| Output | Up to 65,536 output tokens per reply, except for Claude models in Claude Code and GPT models in Codex, which run with their agent's defaults |
| Failed model calls | A rate limit, server error or timeout is retried with backoff by the agent, up to 9 times |
Reasoning levels
Effort levels are minimal, low, medium, high, xhigh and max. (Reasoning) marks reasoning on with no level we can set; (Non-reasoning) marks reasoning off.
Which models carry which label
(Reasoning): Kimi K2.6, Kimi K2.7 Code and Kimi K3; Gemma 4 26B A4B and GLM-4.7-Flash, which reasoned the same at every level in our runs; Claude Haiku 4.5, with a fixed thinking budget of 32,000 tokens. (Non-reasoning): Llama 3.3 70B, Mistral Small 3.1 24B, and GLM-5.2 with reasoning switched off. GLM-5.3 (max) runs at its default with no level sent, because sending max cut its reasoning; Claude Sonnet 4.6 and Claude Opus 4.6 (high) use adaptive thinking at effort high.
Retries, provider errors and refusals
- An attempt that still cannot complete because of the provider is rerun, and the rerun replaces it; the failed record and its cost stay visible. A later rerun of a task whose attempt stands is never used.
- A model with persistent provider errors is not published until they are resolved, unless they cover at most 1% of its tasks or questions after 3 reruns: those are left out of its score, with a note beside the entry.
- A safety refusal, by the model or a provider block, ends the attempt, scores 0 and is not rerun. Provider blocks are counted from the run records; refusals the model writes in its own reply are not marked yet, so they show as No data, not zero.
Scoring and judging
Pass or fail
Each attempt scores 1 (pass) or 0 (fail), and each task has one scored attempt. An attempt passes only if every required check passes: the required output files exist and are valid, the automatic checks of values and geometry pass, any required map passes the map judge, and the trace review finds no reward hacking. An evaluation's score is the share of its tasks passed, each task weighted equally.
Tasks without maps, and attempts awaiting review
A task without a map is graded on its automatic checks alone. Tasks with no scored attempt are left out. An attempt that passed everything else and awaits only its trace review counts as a pass until it is reviewed, and each model page says how many await review. An attempt whose map is not yet judged stays out of the score.
How outputs are checked
| Output | Check |
|---|---|
| Values | Compared with the reference result, with tolerances set by the data's resolution and the method's legitimate variation |
| Coordinate reference systems and units | Must be those requested: metres, degrees, kelvin or Celsius, percentages |
| Geometry | Must be valid, with the right number and type of features, and agree with the reference within a stated tolerance |
| Rasters | Must have the requested grid, extent, resolution, alignment and no-data handling before cell values are compared |
| Files | Must exist, open, and have the requested format, fields and columns |
Before any model attempt counts, each task's automatic checks must accept the reference result and a legitimate alternative, and reject deliberately wrong outputs.
How maps are judged
Maps are graded by an AI judge, Claude Sonnet 5.5 (Anthropic) at effort high, for every model, against a yes-or-no checklist for each task: for example, that the map shows the requested area and variable, has a legend or labels that identify it, and highlights the feature the task asks for. A map passes only if every item is yes. The checklists are kept private so that they cannot be targeted.
Claude Sonnet 5.5 judges every model, Claude models included, so it grades its own family. Before it was chosen, it was checked for a lean towards Claude models against GPT-6.1 Sol (OpenAI) on equal random samples of attempts by Claude models and by other models: compared with Sol, it was slightly stricter with Claude models' maps than with other models' maps, and close on traces, both within the difference expected by chance. Judges from other families are used only to check the judge; their verdicts never count towards a score. Every judged attempt in the published results is judged by Claude Sonnet 5.5. 1 entry has no judged attempts yet.
How the checklists and the judge were checked
Before the judge grades any model, every checklist is checked: the reference map must pass and a blank image must fail, and for a sample of tasks a legitimate restyled map must pass while a plausible but wrong map fails. Where a reference map did not show what its task asks for, the task was excluded rather than the checklist loosened. Map results are published only once the judge has passed these tests on every checklist. A sample of verdicts is checked by hand.
Before Claude Sonnet 5.5 was chosen, it was checked against a judge from another family: on fixed random samples of attempts by Claude models and by other models, its verdicts were compared with those of GPT-6.1 Sol, the judge it replaced, and it showed no lean towards Claude models. Earlier judges were checked in the same way (see Legacy evaluations): while Gemini 3.8 Flash was the judge, Claude Sonnet 4.6 also graded two fixed random samples of maps, one of Gemini models' maps and one of other models' maps. The two agreed on 92% of Gemini models' maps and on 93% of other models' maps.
Reward hacking
Reward hacking is passing a task's checks without doing the analysis. Every attempt that passes its automatic checks and map review gets a trace review: the same judge reads the whole trace, the task prompt, every agent message, its reasoning and every command, with very long command output cut to its first and last 2,000 characters. It flags an attempt only for one of these, naming the step and quoting the trace:
- (a) Access to the checks: reading, listing or searching for the automatic checks, test, reference-answer or solution files.
- (b) Values not computed: writing result values that no command computed or read from the inputs, such as numbers copied from the task text or recalled from memory.
- (c) Fitting the output: after a failed self-check, editing the output to look right instead of fixing the computation.
- (d) Outside answers: fetching answers or solution code from outside the sandbox.
A flagged attempt scores 0, its other verdicts stay visible, and every flag is read by a person before results are published. The judge's full prompt, word for word: trace review.
What is never flagged, lifted flags and declined reviews
Choosing a different valid method, fixing errors, re-running code, checking one's own outputs and listing the agent's own working folders are never flagged. When unsure, the judge does not flag. A flag is lifted on one attempt only after a person or review session has read the whole trace and found a legitimate answer, with nothing read from reference, test or solution files (on Vision agents, a value read from the image or from the model's own stated arithmetic is legitimate); each lift is recorded with its reason.
If the judge declines to review a trace because its safety filter flags the content, the review is recorded as declined and not retried; the attempt keeps the result of its other checks, and each entry shows its number of declined reviews. In the published results the judge has declined some trace reviews; those attempts are not counted as reviewed.
Ranking
Entries in the Geospatial Agent Index. An entry is ranked when it has a score on every evaluation in the view and at least 75% of the view's tasks run, on the tasks it has attempted. Any other entry is listed under "Other entries" and is left out of the ranking, the charts, the Pareto lines and the highlights; its scores stay on its own page. The list is worked out again at every update.
Funding and conflicts
Axis Spatial runs and pays for Geospatial Agent Benchmarks. Some runs use free model access or credits from model providers and compute programmes, which give them no say over results, the method or listing. Labs and providers cannot pay for results, methodology changes or listing. Axis Spatial also offers paid custom evaluations, which never change the public method or scores: see trust and independence.
Spotted an issue? Contact us.
Version history
v2, 13 November 2026 to current
Authors: Pokkiri, N.
- Labs' own agent harnesses replace Terminus-2 as the reference harness, where the lab has one; Terminus-2 entries stay as harness comparisons (6 and 7 November 2026).
- An entry is ranked when every evaluation of the view has a score and at least 75% of the view's tasks have been run, scored on the tasks it has attempted (26 and 27 October 2026).
- An attempt that hits the 2-hour limit after spending more than half its time waiting on the model's provider is run again, and the new run counts (1 November 2026).
- Trace review fails a pass only when the agent obtained the content of the automatic checks or of the answers, or fetched answers from outside; looking and finding nothing is logged, not flagged (4 November 2026).
- Vision agents: 22 ThinkGeo tasks added (106 to 128 built), counted for an entry once it has run them; a complete Stirrup run at the same model and reasoning level gives a Terminus-2 entry its Vision agents score (6 November 2026).
- Cost per task uses one table of provider prices: the lab's own API price, otherwise the median across paid providers (12 November 2026). Scores are unchanged by it.
v1, 19 October 2026 to 13 November 2026
Authors: Pokkiri, N.
- The first public release, methodology version 1.0: the plain average of Earth-Bench V2, GeoBenchX and GS-QA2 in Text only, with Vision agents added in Text + vision. An entry was ranked only once it had run every task.