A good score shows
The synthetic cases can exercise migration-code and guardrail verification.
Geospatial agent benchmark · adapted
Axis Synthetic Migration Cases
Also known as: Axis Synthetic Migration Cases
Evaluates: complete agent systems
The synthetic cases can exercise migration-code and guardrail verification.
At a glance
3Tasks
YesAgents run real code
NoData + code public
YesChecked by recomputation
Axis original syntheticData licence
Sources
every claim on this page traces to these records- Source paper: not published, or not yet verified — we do not guess URLs
- Source data: not published, or not yet verified — we do not guess URLs
- Source code: not published, or not yet verified — we do not guess URLs
- Leaderboard: not published, or not yet verified — we do not guess URLs
In shortAgents run real code inside this benchmark, and answers are checked by recomputation, not by another model's opinion. The underlying data and code are not published at a verified link. Broad migration competence, production readiness, or superiority of one configuration.
Evidence boundaries
what a score on this benchmark does and does not tell youA good score does not show
Broad migration competence, production readiness, or superiority of one configuration.
Arena status
adapted — Only provenance-clear original-synthetic cases may enter a new study; historical results are context, not current ranking truth.
All recorded facts (22 fields)
- Evaluates
- complete-system
- Task families
- migration code generation,guardrail compliance,platform-specific deployment
- Task count
- 3
- Difficulty
- mixed (T1 single-transform through T3 multi-script site suitability with AWS platform constraints)
- Geography
- original synthetic, non-client-specific
- Modality
- mixed
- Input types
- synthetic migration requirements,synthetic source examples
- Output types
- generated Python code with guardrail checks
- Execution environment
- code generation with automated syntax and guardrail checks
- Ground truth method
- AST parse success, guardrail scan pass, prohibited-import absence, dry-run guard presence, and portable-path usage
- Verifier method
- deterministic automated code checks
- Deterministic verification
- yes
- Scoring dimensions
- syntax,guardrails,prohibited imports,dry-run safety,portable paths
- Aggregation formula
- binary verifier outcomes per case and trial; aggregation must be preregistered
- Trials policy
- historical evidence only; pinned repeated trials required
- Variance reporting
- not yet sufficient for a public comparative claim
- Data licence
- Axis original synthetic; publication rights must be recorded per task
- Code licence
- Axis original synthetic; publication rights must be recorded per task
- Contamination concerns
- exclude all historical or uncertain-rights material
- Reproducibility status
- candidate for pinned reproduction
- First published
- 2026-04
- Latest known update
- 2026-04-02
Related Arena records
where this benchmark overlaps Arena's own task packMachine-readable record
This entry is available as JSON at /api/registry#axis-migration-benchmark. See the registry endpoint.