Running a Notification System with BullMQ, Part 3 - Where to Store Delivery History
Where should delivery history live? The trade-offs weighed across a dedicated relational DB, a serial S3 pipeline, SNS fan-out, and a Firehose route.
Blog
Running a Notification System with BullMQ, Part 2 - Rate Limits and the Design They Forced
The sending quota was 100 messages per 5 seconds and our average was 20 req/s. On paper we were inside the limit, yet 429s kept appearing. Two fixed windows whose start times were not aligned turned out to be the cause, and the fact that another team shared the same third-party account pushed us to redesign the sending path itself.
Recompute, Don’t Increment: Updating Review Statistics with RabbitMQ
How I separated review mutations from statistics calculation, then had a Review Consumer recompute statistics from the data available when each RabbitMQ event was processed. This article also covers functional idempotency and the message-loss windows that remained.
Operating a Notification System with BullMQ, Part 1: What Is BullMQ?
Every service needs to send messages to users, whether through AlimTalk, email, or push notifications. But implementing notification delivery as a simple HTTP call quickly runs into limits. Each problem can be solved with another tool: consolidate delivery into one module, enforce a rate limit inside the function, create timing branches with a scheduler, and record delivery…
Why Kubernetes Kills My Pod: Health Probes
Kubernetes provides two kinds of probes for determining a Pod's state. A liveness probe asks, "Is this container alive?" The kubelet periodically calls a configured endpoint and restarts the container if the call fails or receives no response. This automatically recovers an application that has become unresponsive because of a deadlock or memory leak.