Geospatial in Cloud: Databricks

Why Databricks for Geospatial

Before 2023, Databricks was just Spark with a notebook UI. Geospatial meant bolting on Apache Sedona, fighting with UDF serialisation, and hoping your cluster didn't run out of memory mid-join.

That changed. Databricks now has:

The real reason teams choose it: they already have Databricks for analytics. Adding geospatial to an existing lakehouse is cheaper than standing up separate GIS infrastructure. It's incremental compute cost, not new platform cost.

Spatial SQL Deep Dive

The most common geospatial operation is a spatial join: "which parcels fall inside which flood zones?" In Databricks, this is standard SQL.

SELECT
  p.parcel_id,
  p.area_sqm,
  f.flood_zone,
  f.return_period
FROM parcels p
JOIN flood_zones f
ON ST_Contains(f.geometry, p.geometry)
WHERE p.country = 'DE'

No special syntax. No Python UDFs wrapping Java libraries. Just SQL that happens to have spatial predicates. Your data analysts already know how to write this.

A slightly more complex example - buffer analysis. Find all parcels within 1km of each site:

SELECT
  s.site_id,
  COUNT(p.parcel_id) as affected_parcels,
  SUM(p.area_sqm) as total_area
FROM sites s
JOIN parcels p
ON ST_Intersects(
  ST_Buffer(s.geometry, 1000), -- 1km buffer
  p.geometry
)
GROUP BY s.site_id

For a deep dive on the file formats that make this efficient, see our guide on cloud-native geospatial formats (GeoParquet, COG, STAC).

H3 Indexing with Mosaic

H3 is Uber's hexagonal hierarchical spatial index. Think of it as dividing the entire planet into hexagons at multiple resolutions. Every point on Earth maps to a specific hexagon ID at each resolution level.

Why hexagons? Unlike squares, hexagons have a consistent distance from centre to edge in every direction. This matters for spatial joins because it means the index prunes irrelevant partitions evenly, regardless of direction.

The practical difference is dramatic. A spatial join on 100M parcels x 50K flood zones took 47 minutes with ST_Contains. With H3 pre-indexing at resolution 9, the same join took 12 seconds.

from mosaic import enable_mosaic
enable_mosaic(spark)

# Index geometries to H3 resolution 9
df = df.withColumn(
    "h3_index",
    grid_tessellateexplode("geometry", lit(9))
)

# Now spatial joins use H3 index lookups
# instead of full geometry comparisons

Unity Catalog for Rasters

Databricks is vector-first. But most real geospatial workflows involve rasters too - DEMs, satellite imagery, climate grids. Unity Catalog Volumes let you store and version these alongside your vector data.

import shutil

# Stage 1: Copy from Unity Catalog Volume to local SSD
shutil.copy(
    "/Volumes/catalog/schema/rasters/dem.tif",
    "/local_disk0/tmp/dem.tif"
)

# Stage 2: Process locally (random-access works on local disk)
import rasterio
with rasterio.open("/local_disk0/tmp/dem.tif") as src:
    data = src.read(1)
    transform = src.transform

This adds 2-5 seconds of copy time per file, but it's the only reliable pattern. We've filed the issue with Databricks - FUSE filesystem support for random-access reads is on their roadmap but not yet available.

The upside of Unity Catalog: versioning, lineage tracking, and fine-grained access control on your geospatial datasets. One governance layer for everything - vectors, rasters, tabular data. No separate data management for spatial files.

Real Benchmarks

These benchmarks compare the same operations on identical datasets. ArcPy ran on a high-spec desktop. Databricks ran on a 4-node cluster. Both processed GeoParquet data stored on Delta Lake.

The 368x number gets attention, but the real story is cost. That ArcPy run needed a $7,000/year ArcGIS Advanced licence on a $3,000 desktop. The Databricks run cost $0.47 in compute. At 52 weekly runs, that's $24.44/year vs $10,000+/year.

Cost Comparison vs ESRI

Beyond raw performance, the cost difference compounds over time. Here's a side-by-side for a mid-sized geospatial team (20-50 users, 1TB vector data, weekly batch analysis).

When NOT to Use Databricks for Geospatial

This section builds the most trust. We sell Databricks migration services. We still tell clients not to use it in these scenarios:

Getting Started

If you've read this far and Databricks still makes sense for your workloads, here's the minimum viable setup:

%pip install databricks-mosaic
from mosaic import enable_mosaic
enable_mosaic(spark)

# From S3
df = spark.read.format("geoparquet").load(
    "s3://your-bucket/parcels/*.parquet"
)

# From Azure Data Lake Storage
df = spark.read.format("geoparquet").load(
    "abfss://container@account.dfs.core.windows.net/parcels/"
)

# Write to Delta Lake with Unity Catalog
df.write.format("delta").saveAsTable("catalog.schema.parcels")

# Now query with Spatial SQL
spark.sql("""
  SELECT parcel_id, ST_Area(geometry) as area
  FROM catalog.schema.parcels
  WHERE ST_Contains(ST_GeomFromWKT('POLYGON(...)'), geometry)
""")

From zero to running spatial queries: about 30 minutes if you already have a Databricks workspace. Most of that time is waiting for the cluster to start.