Hybrid BigQuery + Spark Architecture

Now we combine structured analytics + large-scale transformation.

? Why Hybrid?

BigQuery:

• Excellent for serving

• Excellent for interactive SQL

• Strong shuffle engine

Spark:

• Better for complex pipelines

• Better for custom ML feature engineering

• Better for non-SQL transformations

? Pattern

Spark Layer

• Heavy ETL

• Data reshaping

• Complex transformations

• Writes parquet / Iceberg tables

BigQuery Layer

• Reads curated datasets

• Provides BI & ad-hoc access

• Hosts feature serving layer

? When to Use Spark Instead of BigQuery

• Extremely skewed operations

• Custom partitioning logic

• Non-relational transformations

• GPU-accelerated ML preprocessing

? Cost Strategy

Spark:

• Scales per job

• Can be cheaper for batch-heavy workloads

BigQuery:

• More efficient for concurrent analytics

• Better slot autoscaling

Hybrid minimizes total cost.