Skip to main content

Lab results · Alibaba

How Alibaba's Qwen models do on geospatial work

Qwen 3.8 27B in Qwen Code is level with Claude Opus 4.7 on Earth observation data. The same model in another harness scores very differently.

Research

Alibaba's Qwen 3.8 27B, running in Alibaba's own coding agent Qwen Code, scores 69 out of 100 on Earth-Bench V2, our benchmark of AI agents analysing satellite and Earth observation data. Claude Opus 4.7 at high reasoning, in Anthropic's own coding agent Claude Code, also scores 69. Qwen 3.8 27B gets there at just over half of Claude Opus 4.7's cost per task.

Each model runs in its lab's own coding agent. Qwen 3.8 27B is an open weights model. Results as of 16 November 2026.

On Earth observation data, Qwen 3.8 27B in Qwen Code is level with Claude Opus 4.7

  1. Claude Opus 5.5 (max)Claude Code80
  2. Kimi K3 (Reasoning)Kimi Code75
  3. GPT-6 Sol (high)Codex75
  4. Kimi K2.7 Code (Reasoning)Kimi Code73
  5. Qwen 3.8 27B (Reasoning)Qwen Code69
  6. Claude Opus 4.7 (high)Claude Code69
  7. MiMo V2.6 Flash (Reasoning)MiMo Code68
  8. MiniMax M3 (Reasoning)MiniMax Code68
  9. Kimi K2.6 (Reasoning)Kimi Code66
  10. Qwen 3.6 35B-A3B (Reasoning)Terminus-262
  11. Qwen 3.6 35B-A3B (Reasoning)Qwen Code59
  12. Qwen 3.8 27B (xhigh)Terminus-254
  13. Claude Haiku 4.5 (Reasoning)Claude Code51
Earth-Bench V2 score, entries in the labs' own coding agents, plus the Qwen models in Terminus-2. Results as of 16 November 2026. Source: Earth-Bench V2.

Which models we ran, and on what

We benchmark two Alibaba models, both open weights: Qwen 3.8 27B and Qwen 3.6 35B-A3B. Each runs in two harnesses. One is Qwen Code. The other is Terminus-2, Terminal-Bench's open harness, our default. Each entry gets one scored attempt per task, in a sandbox with no internet.

The headline score is the Geospatial Intelligence Index, a score out of 100 across three groups of work:

  • Agent tasks (4 of 6 parts): Earth-Bench V2 (satellite and Earth observation data), GeoBenchX (GIS tasks that end in a map, a number or a list) and GS-QA2 (questions about places and terrain, answered by querying a database of OpenStreetMap features and elevation).
  • Guided workflows (1 of 6): GeoAgentBench and GeoAnalystBench, GIS workflows where the task gives the steps and the model writes and runs the code.
  • Geospatial knowledge (1 of 6): questions about GIS and Earth observation, answered with no tools.

41 models are ranked on the index. Qwen 3.8 27B scores 75.8, the highest of the Qwen models, and Qwen 3.6 35B-A3B scores 62.3.

Geospatial Intelligence Index, Text only (0 to 100, higher is better)

  1. Kimi K3 (Reasoning)81.7
  2. Claude Opus 4.7 (high)80.8
  3. DeepSeek V4 Pro (max)79.7
  4. Gemini 3.8 Flash (low)79.6
  5. GLM-5.2 (high)78.6
  6. Qwen 3.8 27B (xhigh)75.8
  7. Nemotron 3 Ultra (high)73.7
  8. Mistral Large 4 (high)71.3
  9. Step 5 Preview (Reasoning)64.6
  10. Gemma 4 26B A4B (Reasoning)64.4
  11. Qwen 3.6 35B-A3B (Reasoning)62.3
  12. MiniMax M3.1 Flash (max)61.3
Both Qwen models, with models from other labs near them. Each model at its best ranked level. Results as of 16 November 2026. Source: Geospatial Intelligence Index.

Where Qwen 3.8 27B is strong

Earth observation data, in Qwen Code. On Earth-Bench V2 the agent works through satellite rasters: vegetation indices, land surface temperature, land cover. In Qwen Code, Qwen 3.8 27B scores 69. Claude Sonnet 5 at high reasoning in Claude Code scores 68 at the same cost per task. Gaps under about 3 points are within our noise.

Following a given workflow. On GeoAgentBench, where the steps are spelled out, it scores 94 in Qwen Code. Claude Opus 4.7 scores 96.

Knowledge. It scores 91 on geospatial knowledge, against 93 for Claude Opus 4.7. The best score of any model is 95.

The harness matters for Qwen

The same model scores differently in the two harnesses. In Qwen Code it scores 69 on Earth-Bench V2; in Terminus-2, 54. On GeoAgentBench the gap is 94 against 62. In the other direction, Terminus-2 scores higher on GeoBenchX, GS-QA2 and GeoAnalystBench, by 4 to 10 points.

Qwen 3.8 27B in Qwen Code against Qwen 3.8 27B in Terminus-2 (0 to 100)

Qwen Code (Reasoning)Terminus-2 (xhigh)

  1. Earth-Bench V26954
  2. GeoBenchX6367
  3. GS-QA27579
  4. GeoAgentBench9462
  5. GeoAnalystBench7989
The same model in two harnesses. The reasoning settings differ, so part of each gap may come from the setting. Results as of 16 November 2026. Source: Qwen 3.8 27B in Qwen Code, every benchmark.

The two entries do not run at the same reasoning setting: Qwen Code runs the model with reasoning on, and Terminus-2 at xhigh. So part of each gap may come from the setting rather than the harness. On the Agent Index, the average of the three agent benchmarks, the two come out close: 69 in Qwen Code and 67 in Terminus-2. Qwen Code costs nearly twice as much per task for those 2 points.

We cover what changes when only the harness changes in Same model, different harness.

Where it falls behind

Knowing when a task cannot be done. GeoBenchX mixes solvable tasks with tasks that cannot be done with the data given, and a correct answer to those is to say so. In Qwen Code, Qwen 3.8 27B answers 38% of the impossible tasks anyway. Claude Opus 4.7 answers 30%. On the solvable tasks Qwen scores 64 against 74, so its GeoBenchX total is 63 against 73.

Spatial questions over a database. On GS-QA2 it scores 75 in Qwen Code and 79 in Terminus-2. Claude Opus 4.7 scores 79. Gemini 3.8 Flash, the best entry in Terminus-2, scores 86.

Qwen 3.6 35B-A3B

Qwen 3.6 35B-A3B is the earlier of the two models. In Terminus-2 it scores 60 on the Agent Index, 7 points below Qwen 3.8 27B in the same harness, at about half its cost per task.

On Earth-Bench V2 in Terminus-2 it scores 62, ahead of Qwen 3.8 27B's 54 in that harness. It is weaker on GIS work: 51 on GeoBenchX, where it answers 63% of the impossible tasks. On the index it scores 56 on guided workflows, against 86 for Qwen 3.8 27B.

What it costs

Cost per task is what one attempt would cost at the provider's list prices, failed attempts included.

Both Qwen models are open weights, so you can also run them on your own servers. Our costs use the list prices of hosted providers; on your own hardware the cost depends on that hardware.

What this means if you are choosing a model

  • Satellite and Earth observation analysis on an open model: Qwen 3.8 27B in Qwen Code is level with Claude Opus 4.7 in Claude Code, at just over half the cost. Put it on your shortlist.
  • Choosing the harness: run Qwen 3.8 27B in both Qwen Code and your own harness on your tasks. The gap between them is large on some benchmarks and goes both ways.
  • Tasks where a wrong answer is worse than none: Qwen answers more impossible GeoBenchX tasks than Claude Opus 4.7. Add a check before its answers are used.

Every score for every model is on the Geospatial Intelligence Index. If you want to know how these models do on a task your team runs today, send it to us and we will benchmark it.