In short
- We benchmark 12 models at the same reasoning level in more than one harness. For 9 of them the scores land within 3 points of each other, which is within our noise.
- Three gaps are large: gpt-oss-20b (high) scores 33 in Terminus-2 and 9 in Codex; DeepSeek V4 Flash (high) scores 72 in Terminus-2, 66 in Codex and 59 in Claude Code; GLM-5.3-Flash (max) scores 72 in Claude Code and 63 in Terminus-2.
- Cost moves more than score. The same model can cost 3 to 10 times more per task in one harness for the same score, and in each of the three big gaps the better harness cost no more.
Why it matters
When you pick an AI model for GIS or Earth observation work, you are picking two things: the model and the harness around it. The harness is the program between the model and your data: it builds the prompt, gives the model its tools, runs its commands and decides when the work is done. Terminus-2, Claude Code, Codex, Kimi Code and Qwen Code are all harnesses. An agent is a harness plus a model.
People often tell us that tool configurations make agent benchmarks hard to compare. The harness is indeed part of the result, which is why every entry on our leaderboard names both model and harness, and we never average across harnesses. Here we ask the next question: hold the model and reasoning level fixed, change only the harness, and how much does the score move?
How we measured
Each entry is one model, at one reasoning level, in one harness. It gets one scored attempt (pass@1) at each of 488 tasks from three published benchmarks: Earth-Bench V2 (analysing satellite and Earth observation data), GeoBenchX (GIS tasks that end in a map, a number or a list) and GS-QA2 (spatial questions over map and terrain data). Every attempt runs in a fresh sandbox with no internet, up to 60 turns and a 2-hour limit; answers are checked by code against the source data, maps by an AI judge, and every passing trace is reviewed for reward hacking. The Geospatial Agent Index is the plain average of the three benchmark scores, and cost per task is what one attempt would cost at the provider's list prices, failed attempts included.
All 12 models here are open weights. Our default harness is Terminus-2 (version 2.0.0), Terminal-Bench's open harness from Harbor, run unmodified for every model. Open models also run in a coding harness: Kimi models in Kimi Code (2.1.1, with one fix that keeps the model's own tool-call names), Qwen 3.8 27B in Qwen Code (0.26.0), DeepSeek and gpt-oss models in Codex (0.162.0) and GLM models in Claude Code (2.1.286). Codex and Claude Code reach these models through a small translator inside the sandbox that converts the harness's requests to the model's own format, with the same settings as the Terminus-2 entry. The full setup is on the agent harnesses page.
What we found
| Model (reasoning level) | Score by harness (cost per task) | Gap |
|---|---|---|
| gpt-oss-20b (high) | Terminus-2 33 ($0.017), Codex 9 ($0.094) | 24 |
| DeepSeek V4 Flash (high) | Terminus-2 72 ($0.065), Codex 66 ($0.085), Claude Code 59 ($0.080) | 13 |
| GLM-5.3-Flash (max) | Claude Code 72 ($0.093), Terminus-2 63 ($0.094) | 9 |
| gpt-oss-120b (high) | Terminus-2 46 ($0.026), Codex 43 ($0.26) | 3 |
| Qwen 3.8 27B (xhigh) | Qwen Code 69 ($0.22), Terminus-2 67 ($0.12) | 2 |
| GLM-5.3 (max) | Claude Code 74 ($0.48), Terminus-2 72 ($0.38) | 2 |
| Kimi K3 (Reasoning) | Kimi Code 75 ($0.32), Terminus-2 73 ($0.23) | 2 |
| GLM-4.7-Flash (Reasoning) | Terminus-2 43 ($0.037), Claude Code 41 ($0.27) | 2 |
| GLM-5.2 (max) | Claude Code 74 ($0.34), Terminus-2 72 ($0.32) | 2 |
| DeepSeek V4.1 Flash (max) | Codex 78 ($0.067), Claude Code 76 ($0.20) | 2 |
| Kimi K2.7 Code (Reasoning) | Kimi Code 72 ($0.18), Terminus-2 72 ($0.11) | 0.5 |
| Kimi K2.6 (Reasoning) | Kimi Code 70 ($0.27), Terminus-2 70 ($0.089) | 0.5 |
Gaps are worked out from the scores to one decimal place, as shown on each model page. "(Reasoning)" marks a model that reasons at a level we cannot set.
Most pairs are within 3 points. For 9 of the 12 models the harness changes the index by less than 3 points. Differences under about 3 points are within our noise, so for these models neither harness is clearly better.
Three gaps are big. gpt-oss-20b (high) scores 33 in Terminus-2 and 9 in Codex, a gap of 24 points. DeepSeek V4 Flash (high) scores 72 in Terminus-2, 66 in Codex and 59 in Claude Code. GLM-5.3-Flash (max) goes the other way: 72 in Claude Code, 63 in Terminus-2. No single harness wins all three. We have not yet traced these gaps to a cause, and we do not claim the harness design alone causes them: in Codex and Claude Code these models run through the translator, and part of a gap may come from that layer.
The lab's own harness is not automatically best. Kimi Code is ahead of Terminus-2 for all three Kimi models, by 2.0, 0.5 and 0.5 points: within noise each time. Qwen 3.8 27B scores 2.5 points higher in Qwen Code, also within noise. And OpenAI's open weights models score lower in OpenAI's Codex than in Terminus-2: 9 against 33 for gpt-oss-20b, 43 against 46 for gpt-oss-120b. Again, gpt-oss reaches Codex through our translator, not OpenAI's own serving.
Cost can differ 3 to 10 times for the same score. Each pair uses the same model at the same token prices, so the difference is in how many tokens the harness uses. gpt-oss-120b costs 10 times more per task in Codex than in Terminus-2, for 3 points less. GLM-4.7-Flash costs about 7 times more in Claude Code than in Terminus-2, for 2 points less; it writes about the same amount (14,000 and 15,000 output tokens per task), so most of the extra cost is input: what the harness sends the model on each turn. DeepSeek V4.1 Flash (max) and Kimi K2.6 each cost three times more in one harness than the other for the same score. In the three big gaps, the higher-scoring harness cost no more.
One level deeper: each benchmark
The index averages three benchmarks, and the harness effect is not the same on each. Qwen 3.8 27B is the clearest case. Its index gap is 2.5 points, but on Earth-Bench V2 (satellite and Earth observation data) it scores 69 in Qwen Code and 54 in Terminus-2, while Terminus-2 is ahead on GeoBenchX (67 against 63) and GS-QA2 (79 against 75).
Earth-Bench V2 often swings most, in both directions: GLM-5.3-Flash scores 69 in Claude Code and 51 in Terminus-2, GLM-5.3 (max) 77 and 67, while DeepSeek V4 Flash scores 73 in Terminus-2 and 56 in Claude Code. gpt-oss-120b scores 55 in Codex against 45 in Terminus-2 on GeoBenchX, but 25 against 42 on GS-QA2.
If most of your work is one kind, look at that benchmark, not only the index. A single benchmark has fewer tasks (117, 150 and 221), so its differences are noisier.
What we don't know yet
- The sample is small. 12 models, all open weights, one run per entry: one scored attempt per task. Our noise estimate covers the choice of tasks, not repeated runs, so real noise is somewhat larger. Claude and GPT models are ranked in their developers' own harnesses only, so we cannot say how they would move.
- Harness versions change. These results are for Terminus-2 2.0.0, Claude Code 2.1.286, Codex 0.162.0, Kimi Code 2.1.1 and Qwen Code 0.26.0. A new version can change a score.
- The plumbing is part of the entry. Open models reach Codex and Claude Code through a translator, and Kimi Code runs with one fix for tool-call names. A big gap may come from that layer, from the harness's own prompts and tools, or from how the model was trained to use them. We have not separated these yet.
- One more pair is coming. DeepSeek V4 Pro (high) in Codex is not ranked yet because its GS-QA2 run is unfinished; on the two benchmarks it has finished, it scores 75 and 72, against 68 and 65 in Terminus-2.
- The common harness is an open question. Today Terminus-2 is the harness most models run in, and our Geospatial Intelligence Index scores each model in its developer's harness where we run one. Whether one common harness or each developer's own is the fairer basis for comparing models is still open; we will write it up when we have an answer.
What to do with this
- Choose the model and the harness together. Read the leaderboard by entry, not by model name.
- If two entries are within 3 points, choose on cost, speed and fit with your workflow.
- Compare cost per task, not price per token. The same model at the same token price can cost ten times more per task in a different harness.
- Look at the benchmark closest to your work. Earth observation analysis and GIS workflows can move in opposite directions.
- Pin your harness version and try a few of your own tasks before you commit: our results are a shortlist, not a guarantee.
Data and how to cite
Every number here is on the live site, with a data table under each chart: the Geospatial Agent Index, the evaluation pages for Earth-Bench V2, GeoBenchX and GS-QA2, and a page for each entry. Scores change as runs complete, so give the date you read them. Method: methodology.
Cite as: Axis Spatial (2026). Same model, different harness: how much does the harness matter for geospatial agents? Geospatial Agent Benchmarks, methodology version 1.0. https://www.axisspatial.com/blog/same-model-different-harness.
Geospatial Agent Benchmarks is an independent, non-commercial leaderboard maintained by Axis Spatial. It is free to read, with no account and no paid placements: we run every model ourselves, on public benchmarks built by others. Questions or corrections: contact us.