Geospatial agent benchmark · not-downloaded

EO-Gym

Also known as: EO-Gym-Data

Evaluates: agents with tool access

Stateful Earth-observation tool use, spatial expansion, temporal retrieval, and cross-modal evidence gathering.

At a glance

9078Tasks
YesAgents run real code
YesData + code public
YesChecked by recomputation
follows the licences o…Data licence

Sources

every claim on this page traces to these records

In shortAgents run real code inside this benchmark, and answers are checked by recomputation, not by another model's opinion. Dataset and code are published, so you can verify claims yourself. A right to mirror the full imagery archive, universal GIS task performance, or production reliability.

Evidence boundaries

what a score on this benchmark does and does not tell you

A good score shows

Stateful Earth-observation tool use, spatial expansion, temporal retrieval, and cross-modal evidence gathering.

A good score does not show

A right to mirror the full imagery archive, universal GIS task performance, or production reliability.

Arena status

not-downloaded — Public paper and paper-linked dataset reviewed; no Arena import or reproduction recorded.

All recorded facts (22 fields)
Evaluates
agent+tools
Task families
optical, multispectral, SAR, temporal, geospatial, disaster, navigation, and counting tasks
Task count
9078
Difficulty
stateful tool use and temporal retrieval
Geography
varied public Earth-observation sources
Modality
mixed Earth observation
Input types
interactive trajectories and indexed public EO files
Output types
tool trajectories and task answers
Execution environment
35 executable Earth-observation tools over more than 660,000 indexed files
Ground truth method
manually verified test trajectories and LLM-validated training trajectories
Verifier method
Pass@k, Tool-any, illegal-call rate, and task-category scoring
Deterministic verification
yes
Scoring dimensions
Pass@k, Tool-any, illegal-call rate, and category scores
Aggregation formula
reported by task category and metric
Trials policy
1,436 manually verified test trajectories; 7,642 LLM-validated training trajectories
Variance reporting
pending verification
Data licence
follows the licences of eight underlying public datasets; no single umbrella licence
Code licence
pending verification
Contamination concerns
underlying public datasets and indexed files require per-source review
Reproducibility status
medium (paper-linked dataset is public; redistribution conditions vary by source)
First published
2026-05
Latest known update
2026-05-02

Machine-readable record

This entry is available as JSON at /api/registry#eo-gym. See the registry endpoint.