Nemotron 3 Ultra (high)

What this metric means. A weighted average of group scores, each 0 to 100: Agent tasks 4/6, Guided workflows 1/6, Geospatial knowledge 1/6.

What this metric means. Closed-book multiple-choice questions on GIS and Earth observation. Geospatial knowledge results.

What cost is measuring. What one attempt at a task would cost if paid token by token at the provider's published API prices, averaged over attempts: ordinary input tokens, input tokens served from the provider's cache at the lower cached price, tokens written to the cache where the provider charges for that, and output tokens (thinking included). Cost assumes each harness uses the model's pay-per-token API, priced at public list rates; sandboxes and our own engineering time are not included. Failed attempts count too.

How to read this chart. Lower is better. Costs are pay-per-token list prices, not any discount or credit a customer might have.

Why a log scale. Costs and times differ by a factor of ten or more between models, so equal steps on the axis are equal ratios.

What cost is measuring. What one attempt at a task would cost if paid token by token at the provider's published API prices, averaged over attempts: ordinary input tokens, input tokens served from the provider's cache at the lower cached price, tokens written to the cache where the provider charges for that, and output tokens (thinking included). Cost assumes each harness uses the model's pay-per-token API, priced at public list rates; sandboxes and our own engineering time are not included. Failed attempts count too.

How to read this chart. Each point is one entry. Further left means a lower average cost per task (log scale); higher means a higher Geospatial Agent Index. Both axes fit the models shown, so the score axis does not start at zero. The shaded most attractive quadrant is the top-left quarter of the plot, split at the middle of each axis: a reading aid that moves with the axes when you change the models shown, not a fixed bar. The dotted Pareto line joins the shown models that no cheaper shown model beats on score.

What this metric means. Time as if every model wrote 100 output tokens per second: output tokens divided by 100, plus the time the agent's commands took. Serving speed differs between providers and is not ranked.

Why a log scale. Costs and times differ by a factor of ten or more between models, so equal steps on the axis are equal ratios.

What this metric means. Time as if every model wrote 100 output tokens per second: output tokens divided by 100, plus the time the agent's commands took. Serving speed differs between providers and is not ranked.

How to read this chart. Each point is one entry. Further left means a lower average time per task (log scale); higher means a higher Geospatial Agent Index. Both axes fit the models shown, so the score axis does not start at zero. The shaded most attractive quadrant is the top-left quarter of the plot, split at the middle of each axis: a reading aid that moves with the axes when you change the models shown, not a fixed bar. The dotted Pareto line joins the shown models that no faster shown model beats on score.

Output tokens per task. The average number of tokens the model wrote per attempt, its reasoning included, as reported by the provider.

Compare with care. Models use different tokenisers, so the same text is a different number of tokens for different models. Use token counts to see how much work each model did, and cost to compare spending.

What this metric means. A turn is one model reply plus the commands it ran.