Item — HN Reader

100%

Every time I work somewhere I have to play shepherd and ask the very basics:

- Who's monitoring queue uptime, setting alerts on it if it goes down, waking up in the middle of the night to fix, patch it, setting it up in all test environments

- Have you thought about all the new problems that might happen: queue sending to dead endpoints, circular queue problem, queue being restarted somehow (e.g. deploys) and losing messages?

- If the app fails post-queue, not surfacing the message to the user, do you have a plan to ensure somebody in engineering sees and fixes that error? And then goes back and remediates the broken request(s)?

- Have you prepared code/logs to do distributed tracing?

- If there's a dispute a week from now whether Joe didn't get an email because of a problem BEFORE or AFTER the queue, will you be able to tell from the logs?

Many powerful engineering abstractions (threads, async, services) require one notch higher of engineering talent and allows for all sorts of new failure paths. The tradeoff must be taken very seriously. Most places I have worked at adopted complexity too soon.

2 comments

Fully agree here. Catastrophic failures can slip in silently on both ends of the curve -- if Joe doesn't get an email OR Joe gets too many emails, it's still bad. In my experience, beefing up logging to be able to do distributed tracing is an absolute must to prevent debugging pain.

Great points! Would you mind elaborating to a relatively inexperienced engineer how the following scenarios can end up happening?

> queue sending to dead endpoints

> circular queue problem

> queue sending to dead endpoints

This is a subcategory of "app fails post-queue." Suppose you build a happy working queue, and someday somebody releases a new version of the queue consumer that isn't backward compatible and therefore is broken. How many messages will be lost on production before this caught? Do you have a way to recover those messages?

> circular queue problem

This is the queue version of an infinite loop. Suppose you have an infinite loop in your code, you'll crash the app but catch it very quickly.

Suppose queue message X calls a function which generates ANOTHER queue message X (infinite loop). This will be VERY HARD to catch and slow down the queue system progressively until its overwhelmed (likely only caught on prod).

Thank you! Sounds like a case where a consumer was updated but producer wasn't.. more room for error for the dev

Interesting point about things getting caught on prod, no amount of stress testing can sometimes reproduce these finicky bugs :)

Logic spread out as different code parts that consumes events and then creates new events can be very hard to follow. A simple loop might be obvious but when part A creates event B and part C reads event B and then creats a new event that part A will read it is harder to find. It may be many steps and also some logic that makes it only happen for specific payloads.

Great explanations by the way but I try to avoid having too much logic spread out over different events.