Cloud-Native Geospatial Formats

What Problems Do Cloud-Native Geospatial Formats Solve?

Cloud-native formats solve three problems: inefficient full-file downloads, analytics platform integration gaps, and data discovery challenges. Traditional formats like Shapefile and GeoTIFF require downloading entire files to access a small region. COG, GeoParquet, and STAC enable HTTP range requests, columnar analytics, and searchable metadata. Before diving into solutions, let's be precise about what we're solving.

The precision access principle: extract only what you need from the whole.

How Does Cloud-Optimized GeoTIFF (COG) Work?

COG reorganises standard GeoTIFF with internal tiling, overviews, and header-first structure, enabling HTTP range requests to read specific pixels without downloading the full file. Reading 1km2 from a 50GB raster goes from 14 minutes (full download) to 2.3 seconds (range request). COG isn't a new format - it's a GeoTIFF organised in a specific way that enables efficient cloud access.

How COG Works

A standard GeoTIFF stores pixels in sequential strips. To read any pixel, you typically need to read from the beginning of the file. A COG reorganises this structure with three key features:

Tile-based access: request only the cell you need, leave the rest untouched.

Creating COGs: What Actually Matters

gdal_translate input.tif output_cog.tif \
  -of COG \
  -co COMPRESS=DEFLATE \
  -co OVERVIEW_RESAMPLING=LANCZOS \
  -co BLOCKSIZE=512

Compression: DEFLATE provides good compression ratios with fast decompression. JPEG is appropriate for imagery where some quality loss is acceptable (70-80% smaller files).

Tile Size: 512x512 can improve performance for large-area queries at the cost of slightly more data transfer for small queries.

Deep Dive: GeoParquet

If COG solves the raster problem, GeoParquet addresses vector data. But it's more than "Shapefile, but faster."

The Columnar Advantage

Shapefiles and File Geodatabases store data row by row. To read any attribute of any feature, you typically scan through records sequentially. Parquet stores data column by column - optimised for analytics queries that access specific attributes, aggregate across records, or filter based on values.

Row vs column: scattered access versus precise selection of what you need.

Creating GeoParquet: Practical Code

import geopandas as gpd

# Read source data
gdf = gpd.read_file("input.shp")

# Write to GeoParquet
gdf.to_parquet(
    "output.parquet",
    compression="snappy",  # Fast decompression
    index=False
)

If you're transitioning from ArcGIS, our ArcPy to GeoPandas translation guide covers the complete workflow conversion process.

Deep Dive: STAC (SpatioTemporal Asset Catalog)

COG and GeoParquet solve access and analytics problems. STAC solves discovery.

STAC is a specification for metadata, not a data format. It defines a standard JSON structure for describing geospatial assets - what they contain, where they are, when they were captured, and how to access them.

Why This Matters at Scale

Consider: 10 years of aerial imagery, satellite data from 3 providers, LiDAR from 5 acquisition projects, and derived products like DEMs. Without STAC, finding relevant data requires knowing folder structures, understanding naming conventions, manually checking date ranges. With STAC, you query a single API:

from pystac_client import Client

catalog = Client.open("https://your-stac-api.com")

# Find all Sentinel-2 imagery for summer 2023
results = catalog.search(
    collections=["sentinel-2-l2a"],
    bbox=[-122.5, 47.5, -122.0, 48.0],
    datetime="2023-06-01/2023-08-31",
    query={"eo:cloud_cover": {"lt": 20}}
)

for item in results.items():
    print(f"{item.id}: {item.datetime}")

When NOT to Use These Formats

Cloud-native formats aren't universally superior. Here's when traditional approaches may be appropriate:

Integration with Modern Data Platforms

The strategic value of cloud-native geospatial formats is integration with enterprise data infrastructure.

df = spark.read.format("geoparquet").load("s3://bucket/parcels/")
df.select(st_area("geometry"), st_centroid("geometry")).show()

SELECT parcel_id, ST_AREA(geometry) as area_m2
FROM parcels_ext
WHERE ST_CONTAINS(aoi_polygon, geometry);

The modern stack: cloud-native formats integrate seamlessly with enterprise data infrastructure.

Migration Strategy: A Practical Roadmap

Migration isn't a weekend project. Here's a phased approach that minimises risk.

The modernisation journey: each step lighter than the last.

Questions to Ask Before You Start