Item — HN Reader

OP here - this type of "checkpoint-based state machine" is exactly what platforms which offer durable execution primitives like Hatchet (https://hatchet.run/) and Temporal (https://temporal.io/) are offering. Disclaimer: am a founder of Hatchet.

These platforms store an event history of the functions which have run as part of the same workflow, and automatically replay those when your function gets interrupted.

I imagine synchronizing memory contents at the language level would be much more overhead than synchronizing at the output level.

3 comments

This is also how our orchestrator (written in Go) is structured. JP describes it pretty well here (it's a durable log implemented with BoltDB).

https://fly.io/blog/the-exit-interview-jp/

Nice! It makes a lot of sense for orchestrating infra deployments -- we also started exploring Temporal at my previous startup for many of the same reasons, though at one level higher to orchestrate deployment into cloud providers.

Yep, though I haven’t used them, I’m vaguely aware that such things exist. I think they have a long way to go to become mainstream, though? Typical Go code isn’t written to be replayable like that.

I think there's a gap between people familiar with durable execution and those who use it in practice; it comes with a lot of overhead.

Adding a durable boundary (via a task queue) in between steps is typically the first step, because you at least get persistence and retries, and for a lot of apps that's enough. It's usually where we recommend people start with Hatchet, since it's just a matter of adding a simple wrapper or declaration on top of the existing code.

Durable execution is often the third evolution of your system (after the first pass with no durability, then adding a durable boundary).

What are the main differences between temporal and hatchet?

The primary difference is that Hatchet is an all-purpose platform for async jobs, so while durable execution is a pattern that we support, we have a lot of other features like concurrency and fairness control, event ingestion, custom queues, dynamic rate limiting, streaming from a background job, monitoring, alerting, DAG-based executions, etc. There's a bit more on this/our architecture here: https://news.ycombinator.com/item?id=43572733.

The reason I started working on Hatchet was because I'm a huge advocate of durable execution, but didn't enjoy using Temporal. So we try to make the development experience as good as possible.

On the underlying durable execution layer, it's the exact same core feature set.