BQ Internal Column Encoding Mechanics

BigQuery uses advanced columnar encoding inspired by Dremel.

? Column Storage

Each column stored separately:

• Broken into column chunks

• Compressed independently

• Metadata contains min/max stats

? Encoding Techniques Used

1. Dictionary encoding (low cardinality)

2. Run-length encoding

3. Bit-packing

4. Delta encoding (for timestamps)

5. Compression (ZSTD-like techniques)

? Why This Matters

When filtering:

BigQuery:

• Reads metadata

• Skips blocks outside range

• Reads only required columns

This is predicate pushdown.

? Nested Field Storage

Nested/repeated fields are stored using:

• Definition levels

• Repetition levels

This avoids joins while preserving hierarchy.

This is why wide nested schemas outperform snowflake schemas.