What functional programmers get wrong about systems

This article, despite clocking in at nearly 10,000 words, appears to have been written in only two days. In fact the author has written five engineering articles in the last week, seeming to average around 5000+ words per article. What a spurt of productivity! Coincidentally, the article happens to be filled with pithy headers and constant sensationally dramatic contrasts that are out of place in long-form technical writing, but which LLMs are known to spam ad nauseum because they are effective for hooking attention in clickbait rags. Curious, that.

The challenges mentioned on the article are challenges that are known to every larger distributed system. They are not challenges solved or to be solved by functional programming.

But the solutions and tools functional programming provides help to have higher verifiability, fewer side effects and more compile-time checks of code in the deployment unit. That other challenges exists is fully correct. But without a functional approach, you have these challenges plus all others.

So while FP is not a one-size-fits-all solution to distributed systems challenges, it does help solve a subset of the challenges on a single system level.

This all sounds very clever, but in practice no, most of the problems are code problems and if you do functional programming you really can avoid getting paged at 3AM because everything broke. (Even if you don't solve the deployment problem the article talks about - you're not deploying at 3AM!). It's like those 400IQ articles about how Rust won't fix all your security problems because it can't catch semantic errors, whereas meanwhile in the real world 70% of vulnerabilities are basic memory safety issues.

And doing the things the article suggests would be mostly useless, because the problem is always the boundary between your neat little world and the outside. You can't solve the semantic drift problem by being more careful about which versions of the code you allow to coexist, because the vast majority of semantic drift problems happen when end users (or third parties you integrate with) change their mind about what one of the fields in their/your data model means. There are actually bitemporal database systems that will version the code as well as the data, that allow you to compute what you thought the value was last Thursday with the code you were running last Thursday ("bank python"). It solves some problems, but not as many as you might think.

I went into this expecting it to be a criticism in the context of low-level systems, and was a bit surprised when it ended up being about distributed. The mismatch the author is describing, I think, is not really about functional programmers so much as everyone not working directly on modern distributed or web software. For being upfront about all the ways distributed programming is different, this is actually one of the best intros I have seen for the non-distributed programmer on what distributed programming is really like.

I still want to believe that future programming languages will be capable of tackling these issues, but exploring the design space will require being cognizant of those issues in the first place.

> Static types, algebraic data types, making illegal states unrepresentable: the functional programming tradition has developed extraordinary tools for reasoning about programs

Looks like the term "functional programming" has been watered down so much that now it is as useful as OOP: not at all.

Look, what matters is pure functional programming. It's about referential transparency, which means managing effects and reason about code in a similar way you can do with math. Static typing is very nice but orthogonal, ADT and making illegal states unrepresentable are good things, but all orthogonal.

An awesome article (if only we remove "functional programmers", feel like they are unfairly called out), hit close to home on operations and pull up a lot of (painful) memories.

Fundamentally - all your system after a certain scale becomes distributed systems, accuracy on a local level is usually not where systemic risks are coming from. A distributed system of fully accurate and correct local programs can implode or explode. Migration, compatibility and system stability are surprisingly dependent on engineer experience till this day.

I often recommend developers learn and use at least one model checking language for this purpose. They’re really good at helping us reason about global properties of systems.

This isn’t unique to Haskell or functional programming at all. I’ve seen programmers from JavaScript to C++ suffer the same fate: tied into the boundaries of a single process they try to verify the correctness of their systems informally. By writing documents and holding long meetings or convincing themselves by prototyping. It’s only a matter of time before the find the error in production.

Hopefully it’s of no real consequence. The error gets fixed and it’s a week or two of extra work. But some times it triggers an outage that causes SLO’s to overshoot and SLAs to trigger which cost the company money. That outage could have potentially been avoided.

But it’s tough getting developers to see the benefit in model checking or any kind of formal methods. The hubris is often too great. It’s not just functional programmers! Although it surprised me when I was working with Haskell developers that so many were resistant to the idea.

(If one has experience and understands functional programming, learning TLA should be rather straight forward)

> If you have a web application with more than one server, you have a distributed system.

Even if you have a web application with only one server, and an embedded database, and no other components, you still have a distributed system. The browser is part of that system.

> Static types, algebraic data types, making illegal states unrepresentable: the functional programming tradition has developed extraordinary tools for reasoning about programs.

But none of these things are functional programming? This is more the tradition of 'expressive static types' than it is of FP.

What about Lisp, Racket, Scheme, Clojure, Erlang / Elixir...

> Your monolith is a distributed system

> If you have a web application with more than one server, you have a distributed system. If you have background job workers, you have a distributed system. If you have a cron job, you have a distributed system. If you talk to Stripe, or send emails through SendGrid, or enqueue something in Redis, or write to a Postgres replica, then you are (I regret to inform you) operating a distributed system.

As soon as you have at least one client talking to your server you have a distributed system.

I feel like Erlang is a functional language that's honest about systems, not just a program. You are almost required to write distributed apps with Erlang, just because it's not weird to spin up thousands of Erlang processes, and you don't have any real way of knowing the order they're going to run so you have to build your applications around async messaging.

It makes me a little sad that Erlang/Elixir is kind of the only platform that does this though...

What did I just read? I liked the bits explaining the basics of the difficulties of versioning data, well known points but explained in a simple way. But then it just kept going without any clear central thesis.

Was the central thesis supposed to be that it's impossible for any tools to ever represent complex migrating systems with well-defined data? Because that's not true. Or that it's not practical or cost-effective to do so? Or was there a central point?

It’s true. But static proof based checks can happen across system boundaries it’s just for historical and stupid reasons we misapply it here.

First put all your services in a monorepo have it all build as one under CI. That’s a static check across the entire system. When you do a polyrepo you are opening up a mathematical hole where two repos can be mismatched.

Second don’t use databases. Databases don’t type check and aren’t compatible with your functional code written on servers.

The first one is stupid… almost everyone just makes this mistake and most of the time unless you were there in the beginning to enforce monorepo you have to live with someone deploying a breaking change that effects a repo somehwere else. I’ve been in places where they simply didn’t understand the hole with polyrepos and they went with it despite me diagramming the issue on a white board (it’s that pervasive). Polyrepos exist for people who like organizing things with names and don’t understand static safety.

And it gets worse. So many people don’t understand that this problem only gets worse over time. The more repos you have in your polyrepo the more brittle everything becomes and they don’t even understand why.

The second one is also stupid but you had to be there in the very very beginning to have stopped it. The inception of the concept of databases should have been created such that any query you run on it needs to be compiled to run on the database and is thus statically checked. This is something that can easily be done but wasn’t done and now the entire World Wide Web just has to make do with testing. Hello? Type checking in application code but not in query code? Why?

That’s like half the problems with distributed systems completely solved. And these problems just exist because of raw stupidity. Stuff like race conditions and dead locks are solvable too but much harder. But these issues are obvious. What’s not obvious are the aforementioned problems that almost nobody talks about even though they are completely solvable.

This is really, really good as an overview of system-level issues. I write only to encourage others to read it.

Ideas that helped me situate many of the issues we struggled with at various companies:

- distinguishing programs (the code as designed) from (distributed) systems (the whole mess, including maintenance and migration, as deployed)

- versioning code (reversibly) and data (irreversibly)

- walk-through of software solutions for distributed systems, and their blind spots wrt data versioning; temporal databases

- semantic drift: schema-less change

- Limitations of type systems for managing invariants

- co-evolving code and data; running old code on old data

The real benefit of systems thinking is recognizing whether an issue is unsolvable and/or manageable (with the current approach, or warrants a technology upgrade or directional shift). It's like O(n) evaluation of algorithms, not for space/time but for issue tractability.

This is one of the main problems I have banging my head on for the past decade, and many of the things mentioned, like Buf, Unison, and more, I only stumbled upon randomly on my own. It's refreshing to read an article on this subject simply because it's so under-discussed. I also wonder to what degree these problems have been solved within the high walls of the tech giants like Google and Amazon.

What? Distributed systems 101 is to write workers that execute stateless, pure, functions and to manage state centrally (basically monads in yet another guise, fan in/fan outs) and to use functional operations to merge and transform data (map reduce).

There are a lot of reasons people don't succeed in systems thinking or in writing good distributed architectures, but functional programming is not one of them. If anything it's an inability to think functionally and to leak state and establish brittle dependencies absolutely everywhere that scuppers systems. Yes, FP has ton's of techniques that aid in good design at lower levels too, that doesn't mean it hampers you at high levels. In fact, much of what this article puts forward is a false dichotomy. None of the lower level FP principles and techniques like strong and expressive types systems are in conflict with the much more general principles of system design covered in the article (and they actually help you realize these principles—think version skew is bad in Haskell? good luck figuring out wtf is even happening in a tangled javascript async promises monstrosity undergoing version skew)

Also, slop, gross. At least edit some of that empty AI verbiage out of the final essay. It has the character and charm of a wet rag and adds zero information.

I really enjoyed this article—I meant to just read the first few paragraphs and skim the rest. I ended up reading every word.

The core thesis sticks with me: existing tooling that helps programmers think about and enforce correctness (linters, compilers, type systems, test suites) operate primarily (and often exclusively) on one snapshot of a project, as it exists at a particular point in time. This makes those tools, as powerful as they can be, unable to help us think about and enforce correctness across horizons that are not visible from the standpoint of a single project at a single point in time (systems distributed across time, across network, across version history).

I feel like the issue of semantic drift is a bigger topic that probably has roots beyond systems architecture/computer engineering.

Also I found the writing style very clear. It helpfully repeats the core ideas, but (IMHO, this is subjective) doesn't do so ad nauseum.

I'm interested in reading other writing from the same author.

This falls super flat for me, because the LLM that wrote this spends all of its time seemingly attacking a straw man.

Outstanding piece. It articulates the actually fundamental correctness problem of most software development:

> “given the actual set of deployments that exist right now, is it safe to add this one?”

This problem has been right in front of our faces for decades, but I haven't ever seen it laid out like this, or fully realised it myself. Which is surprising, given the obvious utility of types for program-level correctness.

I think the next big leap forward in software reliability will be the development/integration of both the tooling to accomplish this, and the cultural norms to use them. Automating this (as far as possible -- they do note that semantic drift is immune to automatic detection) will be as big a deal as the concept of version control for code, or CI/CD.

There's lots here that I need to dig into further.

My problem with functional programming is that hardware is imperative, which means that functional approach isn't "just a different way of thinking" but an extra layer of complexity that needs to explain its existence.

Normally the database and message queue is decoupled from the backend service. This decoupling makes managing simpler and make cross-language things easier.

But if the type system need to cover all these components, then they start coupling again.

Coupling is not necessarily a bad thing as long as it gives good developer UX. If the database is tightly coupled with programming language, then it looks like ORM but better. And it probably can also reduce CRUD biolerplate or N+1 issue etc.

Related, there is SpacetimeDB that make backend run within database, and the backend code is highly coupled with SpacetimeDB's own API

Are there any other libaries / research / etc that sorta take a more functional approach to solving these sort of problems?

For db stuff - what if we flipped the way we write schemas around. A schema is its something that you derive at a current state, rather than starting by writing a schema and then generating migrations? So you can reason over all of it rather than a particular snapshot?

What an amazing article. I feel like my brain has been cleansed of a bunch of old assumptions.

Thank you to the author.

This is an absolutely wonderful article. I helped build a system very similar to the one quoted below:

> A deploy pipeline that queries your orchestrator for running image tags, checks your migration history against the schema registry, diffs your GraphQL schema against collected client operations, and runs Buf’s compatibility checks: this is buildable today, with off-the-shelf parts.

that was largely successful at eliminating interservice compatibility errors, but it felt like we were scrambling around in a dark and dusty corner of software engineering that not many people really cared about. Good to see that there's a body of academic work that I was not entirely aware about that is looking into this.

The article successfully identifies that, yes, there are ways, with sufficient effort, to statically verify that a schema change is (mostly) safe or unsafe before deployment. But even with that determination, a lot of IDLs make it still very hard to evolve types! The compatibility checker will successfully identify that strengthening a type from `optional` to `required` isn't backwards compatible: _now what_. There isn't a safe pathway for schema evolution, and you need to reach for something like Cambria (https://www.inkandswitch.com/cambria/, cited in the article) to handle the evolution.

I've only seen one IDL try to do this properly by baking evolution semantics directly into its type model, typical (https://github.com/stepchowfun/typical) with it's `asymmetric` type label: it makes a field required in the constructor but not the deserialiser. I would like to see these "asymmetric" types find their ways into IDLs like Protocol Buffers such that their compatibility checkers, like Buf's `buf breaking` (https://buf.build/docs/reference/cli/buf/breaking/), can formally reason about them. I feel like there is a rich design space of asymmetric types that are poorly described (enum values that can only be compared against but never constructed, paired validations, fallback values sent over the wire, etc.)

Another one that I think makes a pretty good attempt is the config language CUE and its type system (https://cuelang.org/docs/concept/the-logic-of-cue/), which allows you to do first-class version compatibility modelling. But I have yet to investigate whether CUE is fit for purpose for modelling compatibility over the wire / between service deploys.

With all due respect to Duncan... The reason why this is an issue at all is because the people who architected the internet were not thinking functionally. If they were, they would have come up with something like Nix.

But yes, interfacing between formally proven code and the wild wild West is always an issue. This is similar to when a person from a country with a strong legal system suddenly find themselves subject to the rule of the jungle.

The better question to ask is why did we choose to build a chaotic jungle and how do we get out of it

The version problem is solved well in FP by catching the old function and translating it cleanly. This ofc also works with data representation.

Im not sure what the point of the post is. Minimizing side effects in your code (keeping functions pure) is what gives you the flexibility he says FP misses?!

Terrible article. Nowhere in its litany of what one has to do to build systems that work is anything said about functional programming is incompatible with any of that, or even how the mindset of the functional programmer is incompatible with any of that.

The formula for this article was: truism about X followed by unrelated stuff about Y that the reader is supposed to come away thinking X is incompatible with.

There’s so much in this article where I look at it and I’m like, is this ai slop?

I’m beginning to really appreciate short articles with a few bullet point takeaways.

With respect to version control across systems, when you get into serious stuff where mistakes are measured in lives and/or three commas, there’s just a lot more simplicity in the design of those systems than most people think.

Really big systems often have very simple design principles at their core which are echoed through out the topology.

In secure code like the stuff signal uses, having a hash of the code that is attested by a network of servers and chained back to self-signed identities on the client is the only way to go.

Here’s the hash of the code that’s supposed to be running on my server, here’s the proof that I verified it with all of the hardware and software tools at my disposal from the server, and here’s that hash and attestation embedded into the app on your phone or laptop that’s connecting to my box.

If there’s an easier way to get some semblance of “my device is talking to the right code on the right box,” please enlighten me.

Very thoughtful and well written article.

this website is incredibly unfriendly to scroll on firefox mobile. i could read it. it jumps every time i scroll down

OMG. Sweet relief. This is one of the greatest things I've read in ages.

An FP programmer admitting they live in ivory towers. I never thought I'd see the day.

I love it. HN needs 10 times more of this.

This person has grown up - the rest of you? Get a grip.