- The article retains dated observations on native Spatial SQL, H3, storage, and deployment patterns. Confirm current runtime, preview, and library status before use
- The useful comparison is between workflow patterns, not a universal fastest platform. Query results depend on data, cluster, runtime, partitioning, and implementation
- H3 indexing can change the shape of spatial joins, but resolution, data distribution, edge effects, and validation must be tested on the target dataset
- Delta Lake and open formats can support reproducible spatial data workflows, subject to current platform settings, storage layout, and governance requirements
- Asset Bundles can describe deployment configuration as code; the exact CI/CD and environment contract must be checked for the current project
- The retained cost examples are illustrative. Current pricing requires a dated workload model that includes platform, storage, compute, idle time, and operational ownership
A large parcel workload can expose the limits of a single desktop process. The useful question is not which platform wins in general, but which runtime, data layout, cost model, and review path fit the selected workflow.
This retained post records dated Databricks examples and implementation patterns. Results depend on the dataset, runtime, cluster, storage layout, query, cost model, and validation checks. Treat the benchmark figures as historical evidence to review, not as a current platform promise.
Geospatial in Cloud Series
This is Part 1 of our Geospatial in Cloud series. Each post is self-contained. Part 2 covers AWS. Part 3 covers GCP. Part 4 covers Snowflake. Read the one that matches your stack.
Why Databricks for Geospatial
The retained source material contrasts earlier Spark-plus-library patterns with newer platform features. Runtime, preview, library, and governance status can change; do not treat the contrast as a current platform verdict.
The dated source material describes native Spatial SQL and H3 capabilities in a recent Databricks runtime. Availability, preview status, library dependencies, and function coverage can change. Confirm the target runtime and governance contract before treating these patterns as current.
Native Spatial SQL on Photon (Runtime 17+)
The retained example lists common spatial functions in a native SQL path. Confirm the current function coverage, execution path, serialisation behaviour, and runtime support before implementation. Performance must be measured on the target workload rather than inferred from this note.
Mosaic Library for H3 Spatial Indexing
H3 is a hexagonal indexing pattern that can be implemented through a supported library or native runtime features. The retained timing example is not a current result; test the data volume, resolution, join predicate, and operating cost before choosing it.
Unity Catalog for Data Governance
Versioned geospatial datasets with lineage tracking. One governance model for vectors, rasters, and tabular data. Fine-grained access control and audit logs. No separate data management layer for spatial files.
GeoParquet as the Native Format
Read and write GeoParquet directly from cloud storage (S3, ADLS, GCS). No format conversion step, no Shapefile limitations. Columnar storage means the query engine reads only the columns referenced in your query, cutting I/O costs dramatically.
The real reason teams choose it: they already have Databricks for analytics. Adding geospatial to an existing lakehouse is incremental compute cost, not new platform cost. The spatial SQL functions work on the same tables, with the same permissions, in the same notebooks your team already uses.
Native Spatial SQL (Runtime 17+)
The most common geospatial operation is a spatial join: “which parcels fall inside which flood zones?” In Databricks Runtime 17+, this is standard SQL with spatial predicates. No special syntax, no Python wrappers, no external libraries. Your data analysts already know the language - they just need to learn a handful of spatial functions.
A spatial join uses ST_Contains to test containment. A buffer analysis uses ST_Buffer combined with ST_Intersects. Distance calculations use ST_Distance. The syntax is identical to PostGIS, making migration straightforward for teams with PostGIS experience.
Why Photon matters here: these spatial functions execute on Databricks' vectorised C++ engine. Traditional Spark would serialise geometry objects between Java and Python, losing performance at every boundary. Photon operates on columnar data natively. Databricks reports spatial joins running Reported differences vary by query type and implementation. Compare the target workload with independent, reproducible measurements before selecting a runtime or library.
KEY INSIGHT: MOST TUTORIALS ARE OUTDATED
Search for “Databricks geospatial” and most results still show the old Sedona approach: install the JAR, register UDFs, wrap everything in Python. With Runtime 17+, you skip all of that. Native Spatial SQL works out of the box. If you are starting a new geospatial project on Databricks today, start with native SQL. Only reach for Sedona or Mosaic when native SQL genuinely cannot handle your specific workload.
The dated source material describes a broad set of spatial predicates, measurements, transformations, constructors, and accessors. Confirm the current function set and coverage for the target workload; no single SQL surface covers every enterprise geospatial requirement.
For detail on the file formats that make this efficient, see our guide on cloud-native geospatial formats (GeoParquet, COG, STAC).
Sedona vs Mosaic vs Native SQL
The Databricks geospatial ecosystem has shifted significantly. Understanding what is current and what is legacy saves weeks of going down the wrong path.
| APPROACH | BEST FOR | STATUS | COMPLEXITY |
|---|---|---|---|
| Native Spatial SQL + H3 | Spatial SQL and H3 where supported by the target runtime | Dated note; verify current status | Depends on environment |
| Apache Sedona | Complex geometry operations, spatial R-tree index, existing workflows | Active (independent project) | Medium - JAR dependencies |
| Mosaic | H3 tessellation (was the bridge before native H3) | No longer in active development | Locked to Runtime 13 |
Native Spatial SQL + H3 may be a useful starting point. The right choice depends on workload coverage, runtime support, cluster configuration, governance, and operational constraints. Test spatial joins, buffer analysis, distance calculations, area computations, and H3 indexing on the target environment before selecting a default.
MOSAIC IS NO LONGER IN ACTIVE DEVELOPMENT
If you find tutorials recommending the Mosaic library for H3 operations, check their date and runtime assumptions. The available native H3 function set and Mosaic support depend on the selected release. Confirm the target workspace before choosing a migration path.
Sedona remains a possible option for complex geometry. Operations not available in the selected native SQL path may require Sedona or another library. The retained SpatialBench comparison is dated technical context, not a current price-performance ranking; reproduce it on the target workload.
CRITICAL: SEDONAREGISTRATOR IS DEPRECATED
If you find tutorials telling you to call SedonaRegistrator.registerAll(), that API is deprecated in Sedona 1.5+. The modern initialisation uses SedonaContext. Better yet, skip Sedona entirely for standard operations and use native Spatial SQL. One fewer dependency to manage, one fewer JAR version to track, one fewer thing to break during runtime upgrades.
H3 Indexing (Native, Not Mosaic)
H3 is Uber's hexagonal hierarchical spatial index. Think of it as dividing the entire planet into hexagons at multiple resolutions. Every point on Earth maps to a specific hexagon ID at each resolution level. The native function set, runtime support, and external-library requirement must be checked for the selected workspace and release.
Why hexagons? Unlike squares, hexagons have a consistent distance from centre to edge in every direction. This matters for spatial joins because it means the index prunes irrelevant partitions evenly, regardless of direction.
The practical difference can be significant. H3 can pre-filter geometry comparisons by cell before the target workflow applies exact spatial predicates. The benefit depends on data distribution, resolution choice, cluster size, edge effects, and validation. Measure time and cost on the selected workload; do not reuse the retained speed or cost claims as a general result.
The setup is straightforward: use the native h3_longlatash3 and h3_polyfillash3 functions to tessellate your geometries at the appropriate resolution. From that point forward, spatial joins use H3 cell lookups instead of full geometry comparisons. No Mosaic library needed - these are built into the SQL engine.
Resolution choice matters. Resolution 9 (roughly 174m per hexagon) is optimal for urban analyses - parcel boundaries, building footprints, address-level work. Resolution 7 (roughly 1.2km per hexagon) suits regional analyses - flood zones, administrative boundaries, catchment areas. Going too fine wastes memory on index overhead; going too coarse defeats the purpose of pre-filtering. Most enterprise workloads settle on resolution 8 or 9.
One critical detail on H3 coordinate order: the H3 functions expect (latitude, longitude) order - the opposite of most GIS tools which use (longitude, latitude). This trips some workflows. If a spatial join returns zero matches on data that should overlap, check coordinate order along with CRS, geometry validity, and input filtering.
WHY H3 CHANGES EVERYTHING FOR LARGE JOINS
The retained diagram is a relative complexity illustration. Measure any time or cost change on the target dataset and query; no general reduction is claimed.
WHEN NOT TO USE H3
Point-in-polygon with low cardinality (fewer than 10K polygons) - native ST_Contains is faster because H3 indexing overhead exceeds the join savings. The tessellation step itself has a cost. Only use H3 when your polygon count justifies it.
Unity Catalog for Rasters
Databricks is vector-first. But most real geospatial workflows involve rasters too - DEMs, satellite imagery, climate grids. Unity Catalog Volumes let you store and version these alongside your vector data.
THE TRAP NOBODY DOCUMENTS
Databricks Volumes appear to support file I/O, but _tiffSeekProc: Operation not supported kills any library that tries random-access reads on TIFF files. GDAL, rasterio, and any TIFF-based workflow will fail silently or throw cryptic errors.
The fix: two-stage read.
The fix is a two-stage pattern: copy the file from the Volume to /local_disk0/tmp on the worker node, then open it with rasterio or GDAL from the local path. The /local_disk0/ path is ephemeral SSD attached to the instance - fast reads, proper seek support, wiped on cluster termination. This adds 2-5 seconds of copy time per file, but it is the only reliable pattern.
THIS ALSO AFFECTS READS
GeoPackage reads fail on Volumes too. SQLite creates a -wal (Write-Ahead Log) file even for read operations. The FUSE mount cannot handle this. Stage to local disk first, read, then clean up. This catches everyone the first time - the error message gives no hint that the read path is the problem.
The same two-stage pattern applies on every cloud platform. S3 on AWS, GCS on Google Cloud - all object stores are append-only at the protocol level. They cannot seek backwards in a file. Any format that requires random access (GeoTIFF, GeoPackage, Shapefile) needs local staging first. We cover the AWS and GCP variants in their respective guides in this series.
For long-running jobs, clean up /local_disk0/tmp periodically. The ephemeral disk is finite and will fill up if you process thousands of rasters without clearing intermediate files. The disk is wiped on cluster termination, but during execution it is your responsibility to manage.
The upside of Unity Catalog: versioning, lineage tracking, and fine-grained access control on your geospatial datasets. One governance layer for everything - vectors, rasters, tabular data. No separate data management for spatial files.
Delta Lake for Spatial Data
Delta Lake is the storage layer that makes Databricks geospatial genuinely different from running Spark on raw Parquet files. Three capabilities matter for spatial workloads: physical data layout, time travel, and incremental processing.
Z-ordering by geospatial columns is the first thing to configure. When you run OPTIMIZE with ZORDER BY on latitude and longitude columns, Delta Lake can reorganise Parquet files so that spatially related records are easier to prune. The amount of file skipping and any query improvement depends on data distribution, statistics, partitioning, runtime, and query shape. Measure it on the selected workload; do not reuse the retained percentages or speed range.
Liquid clustering (Runtime 13.3+) is better than Z-ordering. Instead of requiring periodic manual OPTIMIZE runs, liquid clustering continuously reorganises data as it is written. Its operational effect depends on the selected table, runtime, clustering policy, write pattern, and maintenance contract. Test the target update path instead of assuming that it removes optimisation work or matches another layout.
Time travel is the audit trail that regulators love. Every write to a Delta table creates a new version. You can query any previous version by number or timestamp. This can support reproducibility when retention, permissions, source data, code, and output checks are controlled. Confirm the retention and governance settings; do not assume that every historical version is available automatically.
Change Data Feed (CDF) enables incremental processing. Enable CDF on a Delta table and you get a log of every insert, update, and delete. For spatial pipelines that run daily on datasets where a documented subset changes, this can support incremental processing instead of a full reload. The affected-row ratio, correctness checks, retention, and cost benefit must be measured for the target table.
DELTA LAKE SPATIAL OPTIMISATION IMPACT
Deployment with Asset Bundles
The biggest operational difference between a prototype notebook and an operated geospatial pipeline is deployment. Asset Bundles can describe this with a declarative YAML file - databricks.yml - that file that records jobs, cluster configurations, schedules, and environment-specific overrides. Confirm the supported fields and deployment policy for the target workspace.
Instead of manually configuring jobs through the UI (which inevitably leads to drift between environments), you declare the entire pipeline as code. A single databricks bundle deploy command can validate and deploy configuration to a target environment. Test the exact command, permissions, policies, instance types, and rollback path before treating it as an operating procedure.
For geospatial teams migrating from ArcPy, this replaces the "copy the script to the server and hope it works" deployment model. A versioned pipeline can record its cluster configuration, schedule, environment variables, tests, and approval boundary. Automatic deployment still requires a tested CI/CD and rollback contract.
MODULE CACHE: THE TRAP THAT WASTES HOURS
When you update a shared Python module during development, the Spark driver picks up the new code immediately - but the executors still have the old version cached from a previous task. Your driver and workers are running different code versions. The fix is dbutils.library.restartPython() after any module update. For operated jobs, pin module versions in the selected library configuration and test cache invalidation. This reduces one class of mismatch but does not remove the need for runtime checks.
Retained Benchmark Illustration
This retained illustration compares a single-threaded desktop configuration with a distributed cluster configuration. It is not a controlled current benchmark or a fair platform ranking. Re-run the operation on equivalent, documented inputs before using any result in a decision.
CONTEXT ON THESE NUMBERS
Other distributed engines may show a similar contrast with a single-threaded desktop, but this retained material does not rank them. The useful next step is a target run record containing dataset, query, hardware, runtime, cost, and validation method.
RETAINED BENCHMARK ILLUSTRATION - DATED CONFIGURATION
Relative duration illustration
Relative duration illustration
Relative duration illustration
DATED CLUSTER CONFIGURATION
- Cluster: 4x i3.xlarge workers (4 vCPU, 30.5 GB RAM each)
- Runtime: 17 LTS with Photon enabled (native Spatial SQL)
- Data format: GeoParquet on Delta Lake
- ArcPy machine: Retained desktop configuration; hardware and licence assumptions require verification
The retained comparison gets attention, but context matters. It uses a single-threaded desktop against a distributed cluster, so it is not a general platform ranking. A useful current comparison must include equivalent workload definitions, cluster configuration, platform licensing, storage, idle time, and validation effort.
Cost Comparison vs ESRI
The following table is a retained cost illustration for a selected workload. It is not a current quote or a general comparison. A live model must include users, data, compute, storage, licences, idle time, migration, support, and operations.
ESRI STACK
YEAR 1 TOTAL
Model required
YEAR 3 TOTAL
Model required
DATABRICKS STACK
YEAR 1 TOTAL
Model required
YEAR 3 TOTAL
Model required
CAVEAT: MIGRATION ISN'T FREE
Year 1 includes migration effort, access work, validation, training, and operational ownership. Estimate those items for the selected estate instead of reusing a retained week range. For teams evaluating this transition, our ArcPy migration playbook covers the ecosystem evaluation in detail.
When NOT to Use Databricks for Geospatial
This section builds the most trust. Databricks is excellent for the right workloads. But it is the wrong choice in these scenarios:
1. Small or simple workloads
PostGIS on a single server may be simpler for a small workload. Compare cluster startup, job scheduling, storage, support, and operational overhead with a current single-server model.
2. Real-time spatial queries
Databricks is commonly used for batch work. For interactive web queries, measure the required latency and compare a warm PostGIS or dedicated spatial index with the selected Databricks path.
3. Heavy raster processing
Databricks is vector-first. For large-scale raster analysis (satellite imagery time series, DEM processing, multi-band classification), consider Google Earth Engine or a dedicated raster processing pipeline. The FUSE filesystem limitation with TIFFs makes raster-heavy work painful.
4. Teams without existing Databricks
If your organisation doesn't already use Databricks, the overhead of adopting it JUST for geospatial is rarely justified. Platform licensing, training, infrastructure setup - the incremental cost argument vanishes when there is no existing investment. Start with PostGIS or DuckDB Spatial.
5. Desktop workflows that work fine
If an analyst is satisfied with a small desktop workflow, don't migrate for the sake of it. Cloud migration has a real productivity cost during transition. The analyst who knew every ArcGIS shortcut is now a beginner in notebooks. Only migrate when scale demands it.
Getting Started
If you've read this far and Databricks still makes sense for your workloads, here's the minimum viable setup:
1. Cluster Configuration
Select a worker shape, runtime, Photon setting, autoscaling policy, and pool policy from the target workspace and workload model. Record startup behaviour, concurrency, memory pressure, idle cost, and output checks before approving a cluster configuration.
2. Enable Mosaic (optional)
Install the databricks-mosaic package via pip in your notebook, then initialise it on your Spark session only after confirming current package and runtime support. Use a measured target workload to decide whether H3 indexing adds value; do not use a retained record-count threshold as a general rule.
3. Load GeoParquet from Cloud Storage
Read GeoParquet files directly from S3, Azure Data Lake Storage, or GCS using Spark's native GeoParquet reader. No format conversion required. The reader handles geometry deserialisation automatically, and columnar access means you only read the columns your query references.
4. Register as a Delta Table and Query
Write your GeoParquet data to a Delta Lake table registered in Unity Catalog. From that point forward, any spatial SQL query works directly against the table - Spatial functions supported by the target runtime can then be queried from SQL notebooks, BI tools, or programmatic APIs. Confirm permissions, formats, and function coverage before implementation.
Setup time varies with workspace access, cluster policy, data loading, permissions, and validation. A current implementation plan should measure those dependencies instead of reusing the retained setup estimate.
Frequently Asked Questions
Can Databricks handle geospatial data?
Databricks can support vector and other geospatial workflows through platform features and libraries. The exact function set, H3 path, format support, raster behaviour, and scale limits must be checked for the current runtime and data.
Is Databricks faster than ArcGIS for geospatial analysis?
A distributed platform may suit large batch operations, but the retained benchmark is not a general performance claim. Compare the target dataset, hardware, query, runtime, cost, and validation requirements with alternatives such as PostGIS.
How much does Databricks cost for geospatial workloads?
Current cost depends on workload, cluster, runtime, storage, platform pricing, idle time, licences, and operations. Use a dated cost model; do not reuse the figures in the retained example as a current quote.
Databricks is not the right choice for every geospatial workload. Existing platform adoption may reduce some setup work, but the case still depends on measured workload, cost, governance, and operating ownership.
A distributed cloud workflow may change the available scale and operating model. The result is not automatically faster or cheaper. The barrier is often the evidence: target data, runtime behaviour, validation, cost, and operational ownership.
That is what this series is for: dated technical context and questions to test before running geospatial workloads in the cloud.
Get Workflow Automation Insights
Monthly tips on automating GIS workflows, open-source tools, and lessons from enterprise deployments. No spam.