Designing a Global Analytics Control Plane

At 50PB scale, governance becomes more important than compute.

Control plane responsibilities:

• Access governance

• Slot governance

• Dataset lifecycle

• Cost tracking

• Region policy enforcement

• SLA enforcement

? Components

Metadata Layer

• Dataset catalog

• Column classification

• Data lineage

Workload Governance Layer

• Reservation management

• Slot quota enforcement

• Tiered workload classes

Example:

Cost Governance

Every query tagged with:

• Team

• Project

• Business function

• Cost center

Cost allocation must be automated.

? Without Control Plane

50PB environment turns into:

• Cost explosion

• Performance chaos

• Political warfare