Beyond the Hype: Real Architecture, Real Failures

What AI Agents Actually Do

The distinction matters because it determines what you can realistically expect. A chatbot is a conversation partner. An agent is a worker.

The Architecture

This is the pipeline we built. Not a whiteboard sketch - this is running in production. Each stage has a specific job and a specific AI model chosen for that job.

Three architectural decisions drive the reliability of this pipeline.

The Dual-Agent Pattern

This is the insight that changed everything for us. Two agents - one that builds, one that audits - dramatically outperform a single agent doing both.

The improvement after refinement is where the dual-agent pattern really separates itself. A single agent reviewing its own code has the same blind spots that created the errors. The Auditor - running a different model with different training data - catches what the Builder cannot see.

Sandbox Execution

Generating code is the easy part. The hard part is knowing whether it produces correct output. That requires actually running it.

Here is what the Auditor actually validates after every sandbox run:

auditor_validation.py

def validate_output(source_output, generated_output):
    checks = {
        "geometry_count": len(source_output) == len(generated_output),
        "crs_match": source_output.crs == generated_output.crs,
        "schema_match": set(source_output.columns) == set(
            generated_output.columns
        ),
        "spatial_accuracy": source_output.geometry.equals(
            generated_output.geometry
        ),
        "attribute_accuracy": source_output.drop(
            columns="geometry"
        ).equals(
            generated_output.drop(columns="geometry")
        ),
    }
    return checks

Every check is binary. Either the output matches the source or it does not. There is no "close enough" in geospatial processing - a wrong CRS can shift features by thousands of kilometres.

Failure Modes - Honest

About 40% of workflows do not pass on the first attempt. Here is where they fail and why. These numbers come from production runs, not benchmarks.

Real Costs

Everyone claims AI saves money. Here are the actual numbers - including the costs that most vendors conveniently omit.

When AI Agents Make Things Worse

This is the section most AI vendors skip entirely. There are real scenarios where AI agents are the wrong tool. Using them anyway wastes time and money.

What This Means for GIS Teams

AI agents do not replace GIS analysts. They replace the repetitive parts of their work. The distinction is critical because it determines how you plan adoption.

The transition is not instant. Expect 2-4 weeks to identify which workflows are automation candidates, another 2-4 weeks to validate the first batch, and ongoing refinement after that. Teams that skip the assessment phase and try to automate everything at once invariably waste time on workflows that should have stayed manual.

For the business case behind automating GIS workflows, including the cost models and ROI timelines, read our workflow automation guide. For the open-source libraries our agents use for code generation, see the ArcPy to GeoPandas migration guide.