5 posts
Grouping 10,000+ same-category notifications into one third-party group meant the chunks a batch server throws as jobs get consumed asynchronously. Where waiting-children fits in keeping one group and one send on top of that, and how five races were closed one by one and verified with invariants.
Delivery-history JSON sitting in S3 is hard to query as it stands. Four options weighed on always-on infrastructure, the shape of the query cost, and freshness, and why a nightly Parquet conversion won.
Where should delivery history live? The trade-offs weighed across a dedicated relational DB, a serial S3 pipeline, SNS fan-out, and a Firehose route.
The sending quota was 100 messages per 5 seconds and our average was 20 req/s. On paper we were inside the limit, yet 429s kept appearing. Two fixed windows whose start times were not aligned turned out to be the cause, and the fact that another team shared the same third-party account pushed us to redesign the sending path itself.
Every service needs to send messages to users, whether through AlimTalk, email, or push notifications. But implementing notification delivery as a simple HTTP call quickly runs into limits. Each problem can be solved with another tool: consolidate delivery into one module, enforce a rate limit inside the function, create timing branches with a scheduler, and record delivery…