Axis Spatial Spatial Intelligence Index

Search modelsMenu

GS-QA2 Full benchmark leaderboard

Spatial data reasoning over map and elevation data, every admitted question, from GS-QA2 Full. A standalone evaluation: its scores are shown here and on model pages, but it is not counted in any index or ranking until more models have run it.

Score

All

GS-QA2 Full score

Share of tasks passed, 0 to 100 · Higher is better

GS-QA2 Full scoreScores 0 to 100: Claude Opus 5.5 (xhigh), Claude Code: 88; Claude Opus 5.5 (high), Claude Code: 86; Claude Opus 5 (high), Claude Code: 86; Claude Opus 5.5 (medium), Claude Code: 85; Claude Sonnet 5.5 (xhigh), Claude Code: 84; Claude Haiku 5.5 (xhigh), Claude Code: 83; Claude Sonnet 5.5 (high), Claude Code: 80; Claude Haiku 5.5 (high), Claude Code: 79; Claude Opus 4.8 (high), Claude Code: 78; Claude Sonnet 5.5 (medium), Claude Code: 77; Claude Opus 4.7 (high), Claude Code: 77; Claude Haiku 5.5 (medium), Claude Code: 77; Claude Sonnet 5 (high), Claude Code: 76; Claude Haiku 4.5 (Reasoning), Claude Code: 64; Nemotron 3.5 Bolt E4B (Reasoning), Terminus-2: 40255075100Claude Opus 5.5 (xhigh)Claude Code88Claude Opus 5.5 (high)Claude Code86Claude Opus 5 (high)Claude Code86Claude Opus 5.5 (medium)Claude Code85Claude Sonnet 5.5 (xhigh)Claude Code84Claude Haiku 5.5 (xhigh)Claude Code83Claude Sonnet 5.5 (high)Claude Code80Claude Haiku 5.5 (high)Claude Code79Claude Opus 4.8 (high)Claude Code78Claude Sonnet 5.5 (medium)Claude Code77Claude Opus 4.7 (high)Claude Code77Claude Haiku 5.5 (medium)Claude Code77Claude Sonnet 5 (high)Claude Code76Claude Haiku 4.5 (Reasoning)Claude Code64Nemotron 3.5 Bolt E4B (Reasoning)Terminus-24
How to read this chart

What this metric means. The share of tasks the entry passed, 0 to 100.

Data table: GS-QA2 Full score

Open weights

GS-QA2 Full score

Share of tasks passed, 0 to 100 · Higher is better

GS-QA2 Full scoreScores 0 to 100: Nemotron 3.5 Bolt E4B (Reasoning), Terminus-2: 4012344Nemotron 3.5Bolt E4B(Reasoning)Terminus-2
How to read this chart

What this metric means. The share of tasks the entry passed, 0 to 100.

Data table: GS-QA2 Full score
GS-QA2 Full score
ModelDeveloperGS-QA2 Full score
Nemotron 3.5 Bolt E4B (Reasoning), Terminus-2NVIDIA4

GS-QA2 Full: score vs. cost per task

GS-QA2 Full score against average cost per task (USD) · Higher and further left is better

  • Most attractive quadrant
  • Pareto line
GS-QA2 Full: score vs. cost per taskGS-QA2 Full score (0 to 100) against cost per task in US dollars, log scale: Claude Haiku 4.5 (Reasoning), Claude Code: 64.32, $0.054; Claude Opus 5.5 (medium), Claude Code: 85.33, $0.072; Claude Opus 5.5 (high), Claude Code: 86.13, $0.075; Claude Sonnet 5.5 (medium), Claude Code: 77.04, $0.033; Claude Sonnet 5.5 (high), Claude Code: 80.48, $0.037; Claude Haiku 5.5 (medium), Claude Code: 76.7, $0.0040; Claude Haiku 5.5 (high), Claude Code: 79.19, $0.0047; Claude Opus 5.5 (xhigh), Claude Code: 87.82, $0.10; Claude Sonnet 5.5 (xhigh), Claude Code: 83.79, $0.050; Claude Haiku 5.5 (xhigh), Claude Code: 82.61, $0.0065; Claude Opus 5 (high), Claude Code: 85.97, $0.15; Claude Sonnet 5 (high), Claude Code: 75.99, $0.081; Claude Opus 4.7 (high), Claude Code: 76.86, $0.19Most attractive quadrant6065707580859095$0.002$0.005$0.01$0.02$0.05$0.10$0.20$0.50Cost per task (USD, log scale)GS-QA2 Full scoreClaude Opus 5.5 (xhigh), Claude CodeClaude Opus 5.5 (high), Claude CodeClaude Opus 5.5 (medium), Claude CodeClaude Sonnet 5.5 (xhigh), Claude CodeClaude Haiku 5.5 (xhigh), Claude CodeClaude Haiku 5.5 (high), Claude CodeClaude Haiku 5.5 (medium), Claude CodeClaude Opus 5 (high), Claude CodeClaude Sonnet 5.5 (high), Claude CodeClaude Sonnet 5.5 (medium), Claude CodeClaude Opus 4.7 (high), Claude CodeClaude Sonnet 5 (high), Claude CodeClaude Haiku 4.5 (Reasoning), Claude Code
How to read this chart

What cost is measuring. What one attempt at a task would cost if paid token by token at the provider's published API prices, averaged over attempts: ordinary input tokens, input tokens served from the provider's cache at the lower cached price, tokens written to the cache where the provider charges for that, and output tokens (thinking included). Cost assumes each harness uses the model's pay-per-token API, priced at public list rates; sandboxes and our own engineering time are not included. Failed attempts count too.

How to read this chart. Each point is one entry. Further left means a lower average cost per task (log scale); higher means a higher GS-QA2 Full score. Both axes fit the points shown, so the score axis does not start at zero. The shaded most attractive quadrant is the top-left quarter of the plot, split at the middle of each axis: a reading aid, not a fixed bar. The dotted Pareto line joins the points that no cheaper one beats on score. Only exact costs are plotted; the table lists every entry.

Data table: GS-QA2 Full: score vs. cost per task

Token usage

All

GS-QA2 Full: tokens per task

Average tokens per task by type · Lower is better

  • Uncached input
  • Cached input
  • Output (reasoning included)
GS-QA2 Full: tokens per taskClaude Sonnet 5.5 (medium), Claude Code: 63k; Claude Opus 5.5 (medium), Claude Code: 70k; Claude Sonnet 5.5 (high), Claude Code: 72k; Claude Opus 5.5 (high), Claude Code: 74k; Claude Opus 4.8 (high), Claude Code: 77k; Claude Sonnet 5.5 (xhigh), Claude Code: 86k; Claude Opus 5.5 (xhigh), Claude Code: 86k; Claude Opus 5 (high), Claude Code: 97k; Claude Haiku 5.5 (medium), Claude Code: 146k; Claude Haiku 5.5 (high), Claude Code: 162k; Claude Opus 4.7 (high), Claude Code: 167k; Claude Sonnet 5 (high), Claude Code: 182k; Claude Haiku 4.5 (Reasoning), Claude Code: 202k; Claude Haiku 5.5 (xhigh), Claude Code: 206k; Nemotron 3.5 Bolt E4B (Reasoning), Terminus-2: 388k0100k200k300k400kClaude Sonnet 5.5 (medium)Claude Code63kClaude Opus 5.5 (medium)Claude Code70kClaude Sonnet 5.5 (high)Claude Code72kClaude Opus 5.5 (high)Claude Code74kClaude Opus 4.8 (high)Claude Code77kClaude Sonnet 5.5 (xhigh)Claude Code86kClaude Opus 5.5 (xhigh)Claude Code86kClaude Opus 5 (high)Claude Code97kClaude Haiku 5.5 (medium)Claude Code146kClaude Haiku 5.5 (high)Claude Code162kClaude Opus 4.7 (high)Claude Code167kClaude Sonnet 5 (high)Claude Code182kClaude Haiku 4.5 (Reasoning)Claude Code202kClaude Haiku 5.5 (xhigh)Claude Code206kNemotron 3.5 Bolt E4B (Reasoning)Terminus-2388k
How to read this chart

What the parts mean. Uncached input: input tokens read at the full price. Cached input: input tokens served from the provider's cache at the lower cached price. Output: the tokens the model wrote, its reasoning included, as reported by the provider.

Compare with care. Models use different tokenisers, so the same text is a different number of tokens for different models. Use token counts to see how much work each model did, and cost to compare spending.

Data table: GS-QA2 Full: tokens per task
GS-QA2 Full: tokens per task
ModelDeveloperUncached inputCached inputOutput (reasoning included)Total
Claude Sonnet 5.5 (medium), Claude CodeAnthropic5k58k70963k
Claude Opus 5.5 (medium), Claude CodeAnthropic5k64k83870k
Claude Sonnet 5.5 (high), Claude CodeAnthropic5k65k85372k
Claude Opus 5.5 (high), Claude CodeAnthropic5k68k95674k
Claude Opus 4.8 (high), Claude CodeAnthropic6k69k1k77k
Claude Sonnet 5.5 (xhigh), Claude CodeAnthropic7k77k1k86k
Claude Opus 5.5 (xhigh), Claude CodeAnthropic7k77k1k86k
Claude Opus 5 (high), Claude CodeAnthropic7k89k2k97k
Claude Haiku 5.5 (medium), Claude CodeAnthropic8k136k2k146k
Claude Haiku 5.5 (high), Claude CodeAnthropic10k150k3k162k
Claude Opus 4.7 (high), Claude CodeAnthropic7k158k1k167k
Claude Sonnet 5 (high), Claude CodeAnthropic8k173k2k182k
Claude Haiku 4.5 (Reasoning), Claude CodeAnthropic9k190k4k202k
Claude Haiku 5.5 (xhigh), Claude CodeAnthropic12k189k4k206k
Nemotron 3.5 Bolt E4B (Reasoning), Terminus-2NVIDIA374k013k388k

Open weights

GS-QA2 Full: tokens per task

Average tokens per task by type · Lower is better

  • Uncached input
  • Output (reasoning included)
GS-QA2 Full: tokens per taskNemotron 3.5 Bolt E4B (Reasoning), Terminus-2: 388k0100k200k300k400kNemotron 3.5 Bolt E4B (Reasoning)Terminus-2388k
How to read this chart

What the parts mean. Uncached input: input tokens read at the full price. Cached input: input tokens served from the provider's cache at the lower cached price. Output: the tokens the model wrote, its reasoning included, as reported by the provider.

Compare with care. Models use different tokenisers, so the same text is a different number of tokens for different models. Use token counts to see how much work each model did, and cost to compare spending.

Data table: GS-QA2 Full: tokens per task
GS-QA2 Full: tokens per task
ModelDeveloperUncached inputCached inputOutput (reasoning included)Total
Nemotron 3.5 Bolt E4B (Reasoning), Terminus-2NVIDIA374k013k388k

Cost

GS-QA2 Full: cost per task

Average cost per task (USD) by token type · Lower is better

  • Uncached input
  • Cached input
  • Output
  • Not split by token type
GS-QA2 Full: cost per taskValues in US dollars: Claude Haiku 5.5 (medium), Claude Code: $0.0040; Claude Haiku 5.5 (high), Claude Code: $0.0047; Claude Haiku 5.5 (xhigh), Claude Code: $0.0065; Claude Sonnet 5.5 (medium), Claude Code: $0.033; Claude Sonnet 5.5 (high), Claude Code: $0.037; Claude Sonnet 5.5 (xhigh), Claude Code: $0.050; Claude Haiku 4.5 (Reasoning), Claude Code: $0.054; Claude Opus 5.5 (medium), Claude Code: $0.072; Claude Opus 5.5 (high), Claude Code: $0.075; Claude Sonnet 5 (high), Claude Code: $0.081; Claude Opus 5.5 (xhigh), Claude Code: $0.10; Claude Opus 5 (high), Claude Code: $0.15; Claude Opus 4.7 (high), Claude Code: $0.19$0$0.05$0.10$0.15$0.20Claude Haiku 5.5 (medium)Claude Code$0.0040Claude Haiku 5.5 (high)Claude Code$0.0047Claude Haiku 5.5 (xhigh)Claude Code$0.0065Claude Sonnet 5.5 (medium)Claude Code$0.033Claude Sonnet 5.5 (high)Claude Code$0.037Claude Sonnet 5.5 (xhigh)Claude Code$0.050Claude Haiku 4.5 (Reasoning)Claude Code$0.054Claude Opus 5.5 (medium)Claude Code$0.072Claude Opus 5.5 (high)Claude Code$0.075Claude Sonnet 5 (high)Claude Code$0.081Claude Opus 5.5 (xhigh)Claude Code$0.10Claude Opus 5 (high)Claude Code$0.15Claude Opus 4.7 (high)Claude Code$0.19
How to read this chart

What cost is measuring. What one attempt at a task would cost if paid token by token at the provider's published API prices, averaged over attempts: ordinary input tokens, input tokens served from the provider's cache at the lower cached price, tokens written to the cache where the provider charges for that, and output tokens (thinking included). Cost assumes each harness uses the model's pay-per-token API, priced at public list rates; sandboxes and our own engineering time are not included. Failed attempts count too.

How the cost is split. Each part is the entry's average tokens of that type times its list price. The rest, mostly tokens written to the cache, is shown as not split by token type.

Data table: GS-QA2 Full: cost per task
GS-QA2 Full: cost per task
ModelDeveloperUncached inputCached inputOutputNot split by token typeTotal
Claude Haiku 5.5 (medium), Claude CodeAnthropic$0.00082$0.0014$0.00096$0.00082$0.0040
Claude Haiku 5.5 (high), Claude CodeAnthropic$0.00095$0.0015$0.0013$0.00095$0.0047
Claude Haiku 5.5 (xhigh), Claude CodeAnthropic$0.0012$0.0019$0.0021$0.0012$0.0065
Claude Sonnet 5.5 (medium), Claude CodeAnthropic$0.010$0.0058$0.0071$0.010$0.033
Claude Sonnet 5.5 (high), Claude CodeAnthropic$0.011$0.0065$0.0085$0.011$0.037
Claude Sonnet 5.5 (xhigh), Claude CodeAnthropic$0.015$0.0077$0.013$0.015$0.050
Claude Haiku 4.5 (Reasoning), Claude CodeAnthropic$0.0088$0.019$0.018$0.0087$0.054
Claude Opus 5.5 (medium), Claude CodeAnthropic$0.021$0.013$0.017$0.021$0.072
Claude Opus 5.5 (high), Claude CodeAnthropic$0.021$0.014$0.019$0.021$0.075
Claude Sonnet 5 (high), Claude CodeAnthropic$0.015$0.035$0.016$0.015$0.081
Claude Opus 5.5 (xhigh), Claude CodeAnthropic$0.029$0.015$0.030$0.029$0.10
Claude Opus 5 (high), Claude CodeAnthropic$0.033$0.044$0.039$0.033$0.15
Claude Opus 4.7 (high), Claude CodeAnthropic$0.036$0.079$0.036$0.036$0.19
Nemotron 3.5 Bolt E4B (Reasoning), Terminus-2NVIDIANo dataNo dataNo dataNo dataNo data
Claude Opus 4.8 (high), Claude CodeAnthropicNo dataNo dataNo dataNo dataNo data

Speed

All

GS-QA2 Full: time per task

Time per task (at 100 output tokens/s), minutes · Lower is better

GS-QA2 Full: time per taskGS-QA2 Full: time per task: Claude Sonnet 5.5 (medium), Claude Code: 0.2 min; Claude Sonnet 5.5 (high), Claude Code: 0.2 min; Claude Opus 5.5 (medium), Claude Code: 0.2 min; Claude Opus 5.5 (high), Claude Code: 0.3 min; Claude Opus 4.8 (high), Claude Code: 0.4 min; Claude Opus 4.7 (high), Claude Code: 0.4 min; Claude Sonnet 5.5 (xhigh), Claude Code: 0.4 min; Claude Opus 5 (high), Claude Code: 0.4 min; Claude Opus 5.5 (xhigh), Claude Code: 0.4 min; Claude Sonnet 5 (high), Claude Code: 0.4 min; Claude Haiku 5.5 (medium), Claude Code: 0.5 min; Claude Haiku 5.5 (high), Claude Code: 0.6 min; Claude Haiku 4.5 (Reasoning), Claude Code: 0.8 min; Claude Haiku 5.5 (xhigh), Claude Code: 0.9 min; Nemotron 3.5 Bolt E4B (Reasoning), Terminus-2: 4.3 min0 min1.5 min3 min4.5 min6 minClaude Sonnet 5.5 (medium)Claude Code0.2 minClaude Sonnet 5.5 (high)Claude Code0.2 minClaude Opus 5.5 (medium)Claude Code0.2 minClaude Opus 5.5 (high)Claude Code0.3 minClaude Opus 4.8 (high)Claude Code0.4 minClaude Opus 4.7 (high)Claude Code0.4 minClaude Sonnet 5.5 (xhigh)Claude Code0.4 minClaude Opus 5 (high)Claude Code0.4 minClaude Opus 5.5 (xhigh)Claude Code0.4 minClaude Sonnet 5 (high)Claude Code0.4 minClaude Haiku 5.5 (medium)Claude Code0.5 minClaude Haiku 5.5 (high)Claude Code0.6 minClaude Haiku 4.5 (Reasoning)Claude Code0.8 minClaude Haiku 5.5 (xhigh)Claude Code0.9 minNemotron 3.5 Bolt E4B (Reasoning)Terminus-24.3 min
How to read this chart

Time per task. Time as if every model wrote 100 output tokens per second: output tokens divided by 100, plus the time the agent's commands took. Serving speed differs between providers and is not ranked.

Data table: GS-QA2 Full: time per task

Open weights

GS-QA2 Full: time per task

Time per task (at 100 output tokens/s), minutes · Lower is better

GS-QA2 Full: time per taskGS-QA2 Full: time per task: Nemotron 3.5 Bolt E4B (Reasoning), Terminus-2: 4.3 min0 min1.5 min3 min4.5 min6 min4.3 minNemotron 3.5Bolt E4B(Reasoning)Terminus-2
How to read this chart

Time per task. Time as if every model wrote 100 output tokens per second: output tokens divided by 100, plus the time the agent's commands took. Serving speed differs between providers and is not ranked.

Data table: GS-QA2 Full: time per task
GS-QA2 Full: time per task
ModelDeveloperTime per task (min)
Nemotron 3.5 Bolt E4B (Reasoning), Terminus-2NVIDIA4.3 min

All

GS-QA2 Full: score vs. time per task

GS-QA2 Full score against time per task (at 100 output tokens/s), minutes · Higher and further left is better

  • Most attractive quadrant
  • Pareto line
GS-QA2 Full: score vs. time per taskGS-QA2 Full score (0 to 100) against time per task (minutes), log scale: Nemotron 3.5 Bolt E4B (Reasoning), Terminus-2: 3.82, 4.3 min; Claude Haiku 4.5 (Reasoning), Claude Code: 64.32, 0.8 min; Claude Opus 5.5 (medium), Claude Code: 85.33, 0.2 min; Claude Opus 5.5 (high), Claude Code: 86.13, 0.3 min; Claude Sonnet 5.5 (medium), Claude Code: 77.04, 0.2 min; Claude Sonnet 5.5 (high), Claude Code: 80.48, 0.2 min; Claude Haiku 5.5 (medium), Claude Code: 76.7, 0.5 min; Claude Haiku 5.5 (high), Claude Code: 79.19, 0.6 min; Claude Opus 5.5 (xhigh), Claude Code: 87.82, 0.4 min; Claude Sonnet 5.5 (xhigh), Claude Code: 83.79, 0.4 min; Claude Haiku 5.5 (xhigh), Claude Code: 82.61, 0.9 min; Claude Opus 5 (high), Claude Code: 85.97, 0.4 min; Claude Sonnet 5 (high), Claude Code: 75.99, 0.4 min; Claude Opus 4.8 (high), Claude Code: 78.33, 0.4 min; Claude Opus 4.7 (high), Claude Code: 76.86, 0.4 minMost attractive quadrant01020304050607080901000.1 min0.2 min0.5 min1 min2 min5 minTime per task (minutes, log scale)GS-QA2 Full scoreClaude Opus 5.5 (xhigh), Claude CodeClaude Opus 5.5 (high), Claude CodeClaude Opus 5 (high), Claude CodeClaude Opus 5.5 (medium), Claude CodeClaude Sonnet 5.5 (high), Claude CodeClaude Sonnet 5.5 (medium), Claude CodeClaude Sonnet 5.5 (xhigh), Claude CodeClaude Haiku 5.5 (xhigh), Claude CodeClaude Haiku 5.5 (high), Claude CodeClaude Opus 4.8 (high), Claude CodeClaude Opus 4.7 (high), Claude CodeClaude Haiku 5.5 (medium), Claude CodeClaude Sonnet 5 (high), Claude CodeClaude Haiku 4.5 (Reasoning), Claude CodeNemotron 3.5 Bolt E4B (Reasoning), Terminus-2
How to read this chart

Time per task. Time as if every model wrote 100 output tokens per second: output tokens divided by 100, plus the time the agent's commands took. Serving speed differs between providers and is not ranked.

How to read this chart. Each point is one entry. Further left means a lower average time per task (log scale); higher means a higher GS-QA2 Full score. Both axes fit the points shown, so the score axis does not start at zero. The shaded most attractive quadrant is the top-left quarter of the plot: a reading aid, not a fixed bar. The dotted Pareto line joins the points that no faster one beats on score.

Data table: GS-QA2 Full: score vs. time per task

Open weights

GS-QA2 Full: score vs. time per task

GS-QA2 Full score against time per task (at 100 output tokens/s), minutes · Higher and further left is better

  • Most attractive quadrant
  • Pareto line
GS-QA2 Full: score vs. time per taskGS-QA2 Full score (0 to 100) against time per task (minutes), log scale: Nemotron 3.5 Bolt E4B (Reasoning), Terminus-2: 3.82, 4.3 minMost attractive quadrant2.833.23.43.63.844.24.44.64.852 min5 minTime per task (minutes, log scale)GS-QA2 Full scoreNemotron 3.5 Bolt E4B (Reasoning), Terminus-2
How to read this chart

Time per task. Time as if every model wrote 100 output tokens per second: output tokens divided by 100, plus the time the agent's commands took. Serving speed differs between providers and is not ranked.

How to read this chart. Each point is one entry. Further left means a lower average time per task (log scale); higher means a higher GS-QA2 Full score. Both axes fit the points shown, so the score axis does not start at zero. The shaded most attractive quadrant is the top-left quarter of the plot: a reading aid, not a fixed bar. The dotted Pareto line joins the points that no faster one beats on score.

Data table: GS-QA2 Full: score vs. time per task
GS-QA2 Full: score vs. time per task
ModelGS-QA2 Full scoreTime per task (minutes)On the Pareto line
Nemotron 3.5 Bolt E4B (Reasoning), Terminus-244.3 minYes

Score vs. release date

GS-QA2 Full: score vs. release date

GS-QA2 Full score against the model's public release date

GS-QA2 Full: score vs. release dateGS-QA2 Full score (0 to 100) against release date: Claude Haiku 4.5 (Reasoning), Claude Code: 64.32, 15 October 2025; Claude Opus 5.5 (medium), Claude Code: 85.33, 22 September 2026; Claude Opus 5.5 (high), Claude Code: 86.13, 22 September 2026; Claude Sonnet 5.5 (medium), Claude Code: 77.04, 28 September 2026; Claude Sonnet 5.5 (high), Claude Code: 80.48, 28 September 2026; Claude Haiku 5.5 (medium), Claude Code: 76.7, 7 October 2026; Claude Haiku 5.5 (high), Claude Code: 79.19, 7 October 2026; Claude Opus 5.5 (xhigh), Claude Code: 87.82, 22 September 2026; Claude Sonnet 5.5 (xhigh), Claude Code: 83.79, 28 September 2026; Claude Haiku 5.5 (xhigh), Claude Code: 82.61, 7 October 2026; Claude Opus 5 (high), Claude Code: 85.97, 24 July 2026; Claude Sonnet 5 (high), Claude Code: 75.99, 30 June 2026; Claude Opus 4.8 (high), Claude Code: 78.33, 28 May 2026; Claude Opus 4.7 (high), Claude Code: 76.86, 16 April 20266065707580859095Oct 2025Jan 2026Apr 2026Jul 2026Oct 2026Jan 2027Release dateGS-QA2 Full scoreClaude Opus 5.5 (xhigh), Claude CodeClaude Opus 5.5 (high), Claude CodeClaude Opus 5 (high), Claude CodeClaude Opus 5.5 (medium), Claude CodeClaude Sonnet 5.5 (xhigh), Claude CodeClaude Haiku 5.5 (xhigh), Claude CodeClaude Sonnet 5.5 (high), Claude CodeClaude Haiku 5.5 (high), Claude CodeClaude Opus 4.8 (high), Claude CodeClaude Sonnet 5.5 (medium), Claude CodeClaude Opus 4.7 (high), Claude CodeClaude Haiku 5.5 (medium), Claude CodeClaude Sonnet 5 (high), Claude CodeClaude Haiku 4.5 (Reasoning), Claude Code
How to read this chart

How to read this chart. Each point is one entry: across, the date its model was first released; up, its GS-QA2 Full score. Models with no published release date are not plotted.

Data table: GS-QA2 Full: score vs. release date

Background

GS-QA2 Full asks the same questions as GS-QA2, over the same OpenStreetMap vector layers and ASTER elevation data, but runs every question that passed our admission checks, not a balanced sample, across almost all of the release's question templates. The agent gets the question and a PostGIS database in a sandbox without internet access and writes its answer to a file; the check reruns the reference query on the same data with the source's tolerances.

Left out: four templates (two whose answer is a Wikipedia fact not in the data, two that name roads without a place); questions whose reference needs a database too large for the sandbox or ties within 0.5%; 17 questions removed by the audit that also applies to GS-QA2; and the two templates that sum whole features touching a place (V27, V28) until a later run.

Licence: questions and answers ODbL 1.0. Map data © OpenStreetMap contributors, ODbL 1.0. Elevation: ASTER GDEM v3, a product of METI and NASA.

Published by Shang et al. (University of California, Riverside).

Related links

Explore evaluations