Designing a 100PB Multi-Continent Data Platform

At 100PB:

You are no longer designing a warehouse.

You are designing a global data operating system.

? Core Architectural Principles

1️⃣ Data Sovereignty First

Each continent operates a regional data plane:

• North America

• Europe

• APAC

• LATAM

Each region:

• Stores raw data locally

• Enforces residency rules

• Runs compute locally

• Exposes curated exports only

No global raw dataset.

? Two-Plane Architecture

? Data Plane (Regional)

• Object storage (raw)

• Warehouse (regional BigQuery)

• Streaming ingestion

• ML pipelines

? Control Plane (Global)

• Metadata catalog

• Access policies

• Cost governance

• Slot governance

• Lineage tracking

• Chargeback accounting

Control plane is centralized.
Data plane is regional.

? Storage Strategy at 100PB

Break data into:

Only 5–15% of total data should be “hot.”

? Key Insight

At 100PB:

You cannot afford global joins.

Instead:

• Regions compute metrics locally

• Publish aggregated metrics

• Global layer only aggregates aggregates

Never raw.

? Compute Strategy

Each region:

• Dedicated slot reservations

• Independent autoscaling

• Isolated workload classes

No cross-region slot pools.