3 posts
From why columns are stored together, through row groups, chunks, pages and the table of contents at the tail, a walk through the inside of a Parquet file. Seven animations show min/max and Bloom filters pruning row groups and HTTP Range fetching just the needed slices.
Delivery-history JSON sitting in S3 is hard to query as it stands. Four options weighed on always-on infrastructure, the shape of the query cost, and freshness, and why a nightly Parquet conversion won.
Where should delivery history live? The trade-offs weighed across a dedicated relational DB, a serial S3 pipeline, SNS fan-out, and a Firehose route.