Who pays for Geospatial Agent Benchmarks, what can and cannot be bought, how to check our results, and how to tell us when something is wrong.
Independence and conflicts
Axis Spatial runs and pays for Geospatial Agent Benchmarks. Some runs use free model access or credits from model providers and compute programmes, which give them no say over results, the method or listing. Labs and providers cannot pay for results, methodology changes or listing. Axis Spatial also offers paid custom evaluations, which never change the public method or scores: see trust and independence.
Every listed model runs under the same published rules, and the rules for what is run and how it is scored are on the methodology pages.
Paid custom evaluations are benchmarks run for one organisation on its own tasks or data. Paid work never changes the public method, tasks or scores, and its results are shown only with that organisation's consent, labelled as a private evaluation, as the data policy says. To ask about one, contact us.
How to reproduce
What you need to rerun an evaluation yourself is public:
- Tasks: each evaluation's source benchmark, version and licence, and how we chose and adapted its tasks: task selection, with every task left out and why in excluded tasks.
- Harnesses: each agent harness and the version we run: agent harnesses. Runs use Harbor, an open framework for running agent benchmarks.
- Run settings: the sandbox, turns, time limit, reasoning levels, temperature and retries, on the Geospatial Agent Index methodology page.
- Judging: the trace judge's prompt, word for word, on the trace review page, and how maps are judged on the Geospatial Agent Index methodology page.
What stays private, as the data policy says: the code that checks each task, reference answers, held-out tasks and agent transcripts. Get source benchmark data from its authors, under its licence. So a rerun can match our method and settings, but not our grading code.
Report an error
Spotted an issue? Contact us. Corrections to published scores are listed on the updates page, and changes to the method in each index's version history.
External review
There has been no external review yet. When someone outside Axis Spatial has reviewed the method, they are named here, with their consent.