ImpactMesh benchmark leaderboard
Wildfire and flood mapping from satellite images at four dates, from ImpactMesh. A standalone evaluation: its scores are shown here and on model pages, but it is not counted in any index or ranking until more models have run it.
Models90 of 90 entries
Score
ImpactMesh score
Share of tasks passed, 0 to 100 · Higher is better
No data yet.
How to read this chart
What this metric means. The share of tasks the entry passed, 0 to 100.
Data table: ImpactMesh score
Background
ImpactMesh pairs Sentinel-2 optical and Sentinel-1 radar images, taken at four dates around a wildfire or a flood, with the burn scars and flood extents mapped by the Copernicus Emergency Management Service. Here the agent gets one 7.68 km scene of an event and works out the burned or flooded area, measures the change between two dates by a procedure the task sets out, or draws the burn scar or flood as a map.
We run a small slice: 12 tasks on four scenes from the dataset's test split, two wildfires and two floods. Ten of the candidate events we checked were left out: five whose images were taken on different dates across the scene, and five where a standard burn index or radar water method cannot reproduce the emergency map from these images. So this set is easier than ImpactMesh as a whole.
Licence: dataset CC BY 4.0; contains modified Copernicus Sentinel data, Copernicus DEM (© DLR e.V. and Airbus) and Copernicus Emergency Management Service maps (© European Union).
Published by IBM, DLR and ESA Φ-lab.
Related links
Explore evaluations
- Geospatial Intelligence IndexAgent tasks, vision, guided workflows and geospatial knowledge, per model
- Geospatial Agent IndexEarth-Bench V2, GeoBenchX and GS-QA2; Text + vision adds Vision agents
- Earth-Bench V2Earth-Bench V2 poses Earth observation questions over satellite products (indices, temperatures, land cover) that are answered by computing over the data.
- GeoBenchXGeoBenchX asks for maps, numbers and lists from country-level and city-level geospatial data.
- GS-QA2GS-QA2 asks questions about places and terrain in the United States over OpenStreetMap vector layers and ASTER elevation data: nearest features, directions, distances, counts, elevations, slopes and features that meet terrain conditions.
- GeoAgentBenchGeoAgentBench gives an agent a geospatial question and the data to answer it, and expects the result files and a map.
- GeoAnalystBenchGeoAnalystBench collects GIS analyst workflows (suitability analysis, interpolation, hot spots, network and overlay work).
- OGD4AllOGD4All asks everyday questions about the City of Zurich's open government data: trees, buildings, transport, schools and more, from about 430 public datasets in GeoPackage and CSV form.
- GeoNatureAgentGeoNatureAgent asks for figures from environmental rasters over Spanish provinces and regions: here, the share of land suitable for CO₂ storage, by province, with follow-up questions that compare, rank or break the figures down.
- MultiGlobeQAMultiGlobeQA asks spatial questions about places all over the world: what lies within, next to or nearest to what, how far apart things are, and how many there are.
- GeoAgentEvalGeoAgentEval tests whether an agent can build, audit and repair spatial data in a PostGIS database, and decline honestly when a request cannot be done: flood zones, wards, routes, coordinate systems and stale records in one made-up town.
- GeoChronoGeoChrono asks about long-term change in a place seen over many years: what the land cover is, whether an area grew or shrank, when an object appeared and how things changed in order.
- BigEarthNet.txtBigEarthNet.txt asks about 1.2 km image patches over Europe: which land cover is present, how much, how many patches of it, what is next to what, and the country, season and climate zone.
- Geospatial knowledgeClosed-book multiple-choice questions on geospatial knowledge
- VisionQuestions answered from images
- Earth-Bench V2 imagesEarth-Bench V2 images are the Earth-Bench V2 questions that need the model to look at RGB satellite pictures: multiple choice on counting objects or scene types, ranking pictures by count or area, and comparing what changed.
- ThinkGeo imagesThinkGeo asks for one number about aerial and satellite images: a count, an area, a distance or a heading, accepted within a range.
- Vision agentsAgents that look at aerial images and use code