Full setup
A result records the model, prompt, tools, retry policy, task set, data, and verifier version. Model names alone do not make results comparable.
Research protocol
How the Arena checks a run
Arena records the task, model, output, checks, time, cost, and failures. A result is a reproducible chain of inputs, complete configuration, execution, artefacts, verifier checks, recovery behaviour, and candid failure states.
Four pillars
what makes a result usableVerifier first
Verifiers inspect the output and recompute expected values where possible. Empty files, fabricated numbers, unsafe spatial operations, and shortcuts fail.
Run record
Each record keeps the output, attempts, logs, artefacts where available, and verifier detail. Reproducibility is a reported condition, not an assumption.
More than completion
A result also shows cost, latency, retries, false-success rejections, and reproducibility. A fast plausible answer is not accepted work.
Status vocabulary
one meaning per word, used everywhereAccepted
The verifier's deterministic checks all passed for the final attempt within the task budget.
Rejected
The output exists but failed one or more independent checks. The failing check names expected and actual values.
Blocked
No executable attempt was produced: an inaccessible model, a sandbox start failure, or an empty program.
Inconclusive
A run mixed accepted and non-accepted trials. Headline scores quote the accepted subset and list every run beside it.
Cancelled or expired
A run stopped before completing its planned trials, or exceeded its wall-clock budget. Partial rows remain visible.
Common failures
a plausible map can still be wrong · worked examples linkedOutput and area
Empty output, area drift, unsafe buffers, and raster misalignment can look complete while failing the ground truth.
Spatial meaning
Wrong CRS, units, nodata handling, or cloud masks are correctness failures, not style differences.
Fabrication
Invented statistics, hardcoded expected values, or outputs written without reading inputs. Independent recomputation catches these.
Evidence
An unrepeatable result cannot support acceptance, even when the output looks convincing.
Worked examples
Run ea17d555 shows accepted lanes beside rejected ones. Run c55f0f7f shows a mixed outcome kept visible as inconclusive rather than averaged away.
Measurement rules
Variance policy
Every configuration runs at least three trials per task before any reliability claim. Single observations are labelled as such. Pass-at-k is reported when retries are allowed.
Judgement models never decide
A judgement model can rate plausibility but cannot prove geometric correctness. Its scores never contribute to acceptance and appear, if at all, as descriptive context next to deterministic results.
Snapshot sealing
When a run finishes, its comparison table is written once as a content-addressed snapshot. Published figures read from sealed snapshots so they cannot drift afterwards.
Current records
Arena publishes only current records that can be inspected. Incomplete records are not used for comparisons.
Glossary
terms used across run and evidence records- Pack version
- The pinned set of task definitions a run executed. Results from different pack versions are never compared directly.
- Configuration
- A complete comparison unit: model, prompt version, tools, retry policy, and pinned task, data, and verifier versions. A model name alone is not comparable.
- Lane
- One configured route through which a model is reached. Duplicate upstream models across lanes count as adapter evidence, not extra capability.
- False-success rejection
- A trial whose output looked plausible but failed independent deterministic checks, such as an empty file or a fabricated value.
- Token-cost proxy
- Cost estimated from reported token usage and list prices at recording time; actual billing may differ.
- Sealed snapshot
- An immutable, content-addressed copy of a leaderboard computed from a finished run.
Frequently asked questions
definitional answers, kept shortWhat is a geospatial agent benchmark?
- A test set plus a checking protocol for AI systems that perform geographic information system work: reading spatial data, writing code, and producing maps or statistics. A useful benchmark states what it measures, how answers are verified, and what its scores do not prove.
Can an LLM judge prove a GIS agent's output is correct?
- No. A judge can flag obviously implausible output but cannot independently recompute geometry, areas, or statistics. Acceptance here always comes from deterministic checks against recomputed expectations; judge opinions are side metrics at most.
Which benchmarks execute agent code?
- Only benchmarks with a real execution environment can show that generated code runs and produces its stated output. The comparison table marks execution per benchmark, including where evidence is pending verification.