Satellite imagery
84tasksOptical multispectral
Satellite bands, thermal bands and indices such as NDVI
Earth-Bench V2 84
120 questions in BigEarthNet.txt, outside the index.
Tasks
The geospatial tasks the agents are given: where on Earth they are set, what data they read, and how the models score on each kind of work, task format and task length. These groupings do not change the indexes.
Where the tasks are
Each circle is a place one or more tasks are set, its area the number of tasks there; outlines are larger areas. 996 of 1136 agent tasks have a place, 41 cover the whole world and 99 have no location.
- Earth-Bench V2114 of 117
- GeoBenchX95 of 150
- GS-QA2221 of 221
- Guided workflows65 of 69
- Vision agents11 of 64
- OGD4All182 of 182
- GeoNatureAgent14 of 14
- MultiGlobeQA294 of 294
- GeoAgentEval0 of 25
Circle area: tasks at a place110100
Data table: tasks by evaluation and location
| Evaluation | Tasks | At a place | Worldwide | No location |
|---|---|---|---|---|
| Earth-Bench V2 | 117 | 114 | 0 | 3 |
| GeoBenchX | 150 | 95 | 39 | 16 |
| GS-QA2 | 221 | 221 | 0 | 0 |
| Guided workflows | 69 | 65 | 2 | 2 |
| Vision agents | 64 | 11 | 0 | 53 |
| OGD4All | 182 | 182 | 0 | 0 |
| GeoNatureAgent | 14 | 14 | 0 | 0 |
| MultiGlobeQA | 294 | 294 | 0 | 0 |
| GeoAgentEval | 25 | 0 | 0 | 25 |
| All evaluations | 1136 | 996 | 41 | 99 |
Circle area: tasks at a place110100
Circle area: tasks at a place110100
Circle area: tasks at a place110100
Not drawn
41 tasks cover the whole world
- World: 38
- Arctic countries: 2
- G7 and BRICS countries: 1
Not drawn
99 tasks have no place
- Vision agents: 53, as their images carry no coordinates and name no source with a known place (only xBD images name theirs: the disaster they record)
- GeoAgentEval: 25, as its database is a made-up town, whose coordinates fall in open sea, and the task names no real place
- GeoBenchX: 16, as the task names no place (the data cannot answer it)
- Earth-Bench V2: 3, as its rasters carry no coordinates and the question names no place
- GeoAgentBench: 2, as its data carry no coordinates and the task names no place
How the footprints are found
Where a footprint comes from. Earth-Bench V2 and guided workflows: the extent of the task's own input data, or the place the task names when its data carry no coordinates. GS-QA2: the place the question names (never its answer). GeoBenchX: the country or region the task names, as the box around those countries' main land areas. Vision agents: the area of the disaster its xBD images record (the states a hurricane or flood struck, or the place a fire or eruption hit), not the image's own place; the other images carry no coordinates.
Two places. A task whose data lie in two distant places, such as two cities compared, has a footprint at each.
What the tasks read
Data types
Tasks of each data type; the table gives each entry's score, 0 to 100 · Higher is better
Maps and GIS layers
186tasksVector only
Points, lines and polygons, with attribute tables
GeoBenchX 73 · GS-QA2 113
294 tasks in MultiGlobeQA, 171 in OGD4All and 25 in GeoAgentEval, outside the index.
Aerial photos
64tasksOptical RGB, very high resolution
True-colour satellite or aerial photographs
Scored in Vision agents (Text + vision view) and on Vision, so not in the score table.
300 questions in GeoChrono and 120 in BigEarthNet.txt, outside the index.
Radar (SAR)
120tasksSAR
Radar backscatter imagery
120 questions in BigEarthNet.txt, outside the index.
Land cover, population and lights
45tasksOther gridded data
Population, land cover, night lights and other grids
Earth-Bench V2 37 · GeoBenchX 8
14 tasks in GeoNatureAgent, outside the index.
Weather and climate
10tasksClimate or gridded time series
Climate grids, NetCDF fields and time series
Earth-Bench V2 3 · GeoBenchX 7
Terrain
108tasksElevation or terrain
Elevation models and surfaces derived from them
GS-QA2 108
No tasks yet
Places and coordinates
Finding a place from its name or its coordinates, and the distance and bearing between places. No tasks yet.
Data table: Data types
A task counts under every data type it reads.
Models87 of 87 entries
By kind of work
What the agent has to do: the main kind of geospatial work in each task.
All
Remote sensing and raster analysis score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Remote sensing and raster analysis score
Open weights
Remote sensing and raster analysis score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Remote sensing and raster analysis score
All
GIS analysis score
Share of tasks passed, 0 to 100 · Higher is better
Data table: GIS analysis score
Open weights
GIS analysis score
Share of tasks passed, 0 to 100 · Higher is better
Data table: GIS analysis score
All
Climate and time series score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Climate and time series score
Open weights
Climate and time series score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Climate and time series score
All
Spatial statistics score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Spatial statistics score
Open weights
Spatial statistics score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Spatial statistics score
All
Making maps score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Making maps score
Open weights
Making maps score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Making maps score
All
Recognising infeasible requests score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Recognising infeasible requests score
Open weights
Recognising infeasible requests score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Recognising infeasible requests score
- Remote sensing and raster analysisSatellite band maths, reclassification, terrain and raster overlay, 181 tasks
- GIS analysisBuffers, spatial joins, overlay and dissolve, 118 tasks
- Climate and time seriesSeries over many dates and gridded climate data, 59 tasks
- Spatial statisticsKriging, density, hot spots, regression and clustering, 13 tasks
- Making mapsMaps as the main deliverable, 57 tasks
- Recognising infeasible requestsSaying no when the data cannot answer, 60 tasks
By task format
Whether a task asks for one answer, a multi-step analysis, or for the agent to see that the request cannot be done.
All
Multi-step analysis score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Multi-step analysis score
Open weights
Multi-step analysis score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Multi-step analysis score
All
Single-answer questions score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Single-answer questions score
Open weights
Single-answer questions score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Single-answer questions score
All
Recognising infeasible requests score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Recognising infeasible requests score
Open weights
Recognising infeasible requests score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Recognising infeasible requests score
By task length
How long a task takes: the median time of every agent's attempts at it.
All
Short tasks score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Short tasks score
Open weights
Short tasks score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Short tasks score
All
Medium tasks score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Medium tasks score
Open weights
Medium tasks score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Medium tasks score
All
Long tasks score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Long tasks score
Open weights
Long tasks score
Share of tasks passed, 0 to 100 · Higher is better
Data table: Long tasks score
Median time per task at 100 output tokens/s: short up to 44 seconds, medium up to 1.6 minutes, long more than 1.6 minutes.
The same attempts as the Geospatial Agent Index. See the methodology for how each task is assigned.