This is one of my favorite blog posts, and it can basically be encapsulated in the idea of "innovation tokens." It is one of the most useful concepts I have had as a PM / eng leader in my career. It helps actually make the the right tradeoffs, and helps even more in explaining those tradeoffs to colleague of all levels. Highly recommend.
But innovation tokens is a neat idea, it's not personal - no big bad grumpy old senior being a square about CoolTech2026 - and it's an incentive to have someone that wants to do something cool think about it. The other one in that regard are ADRs, which I think is a very low barrier to entry path into architecture that most developers can do. It too forces the person to consider CoolTech2026 in a wider context - what does it solve, what is used right now to solve it, what other solutions are out there, etc - plus making it a team decision if executed right.
Maybe it wasn't beans? But, I've been looking for it for years.
> If you choose to write your website in NodeJS, you just spent one of your innovation tokens.
Is NodeJS still considered on the same level of "unknown" / "not-boring" technology as the others listed? By my reckoning it is plenty mature enough to be considered a "boring" choice, and going for bun would be spending and innovation token.
The amounts clearly aren't discrete. Writing your entire app on a new language runtime might be very risky (and also might have large potential ROI!), but choosing a new email provider might not be (random example, but presuming that you can swap out providers fairly easily).
[0] My bigger complaint is really about the vagueness of even deciding what is "boring." How does a new technology transition from being "non-boring" to being "boring"? Apparently that requires a lot of people to ignore this article's advice for a long time, until we collectively decide that those people have had good enough results to consider that technology "boring."
But .... if you run a multi cloud hybrid setup with kubernetes, service mesh, data [lake|pond|ocean] and millions of other fancy words in tech at each and every layer, you resume would look so awesome and you sound wicked smart. And the VP gets 600M budget for AWS and 600 developers, SREs, DevOps, PMO. It is not that things won't run, humans have perverse incentives.
I'm certain a lot of porn/adult industry run their setup like I mentioned with a Romanian dude running the entire infrastructure for $15K - $20K.
I think that a lot of devs just think they _have_ to do it that way, or they over-optimize too early and build the ultimate system before they even know if anyone will use the app or site.
Today x86 hardware is super reliable and unneeded complexity in the software stack is the main source of problems.
I agree. Simple and effective.
I would add a third small machine to give HAProxy a third vote. The problem with only two machines is that, if communications between them fail, you don't want both believing the other died and doing the wrong thing. You could use that third machine for Postgres backups, which would be smart because Postgress replication is not the same as backup.
Engineers should understand requirements, risks, tradeoffs, and potential gains. New technology may be right for that. Novel approaches may be right for that. "Novel" or "New" are only proxies and they're weak.
For example, I may think "New" means untested, but is that true? What if a new project has Jepsen testing, a fuzzing suite, massive compute running tons of oracle tests, etc? I should just say "Choose well tested" instead of "Choose old" - lots of old software is very poorly tested.
Maybe I think that "Old" implies better documentation, but does it? Lots of older projects have insane cruft and weird edge cases that are undocumented and accumulated over years.
Why do we need a metaphor? Why is "innovation token" helpful?
If you're incapable of evaluating a technology in terms of these properties, you aren't a serious developer and "boring" will not save you.
Sit down, write our your requirements, determine candidate solutions, and choose them based on their fit. "Boring" means nothing, it's a vague proxy term. "Well tsted", "performant for our use case", "developers know it", etc mean something.
> MySQL is boring. Postgres is boring. PHP is boring. Python is boring. Memcached is boring. Squid is boring. Cron is boring.
Literally every one of these has caused hilarious and disastrous failures for me in my career. But yep, boring.
> If you choose to write your website in NodeJS, you just spent one of your innovation tokens. If you choose to use MongoDB, you just spent one of your innovation tokens.
What if you know NodeJS really well? Or MongoDb? What if you have empirical, verifiable reasons for why they fit better?
I'm a bit tired of "simple" and "boring" and other nonsense words in this field taking up the air in the room that should be spent evaluating solutions on their actual merits.
On the other hand, IBM was late getting into integrated circuits. They had Solid Logic Technology, automated machinery for putting transistors into ceramic substrates to make tiny but discrete circuits. That's what powered the IBM System/360. Worked, but kept mainframe prices high and made IBM late to minicomputers.
Innovation when the problem is hard is more interesting. Look at the history of US long range bombers. The B-29 was effective, but underpowered, and had a lot of trouble getting off the ground fully loaded. So, after WWII, the next development was the B-36, which answered the question "what if we scaled up the B-29?"[1] Six propellers, and four jet engines (added late in the design cycle). Was a sky barge, but it worked, as long as no one was trying hard to shoot it down. No flyable aircraft remain. That was the boring technology approach.
After that came the B-47, which answered the question "what if we scaled up a jet fighter to bomber size?"[2] The B-47 was all jets, no props. It needed solid-fuel rocket boosters (!) to help it get off the ground, and a drag chute to slow it down on landing. It was terrible to fly; its operating speed and altitude were too near the "coffin corner" where stall and Mach buffet meet.[3] No flyable aircraft remain.
Then came the B-52. New engines. New airframe. New design. [4] That's the high-risk approach. It worked. Hundreds are still in active service. Outlasted most of its successors, the B-58 Hustler (a supersonic intercontinental bomber), the B-70, the F-111, etc.
On the commercial side, we have Boeing, whose most successful airliner is a variant of the B-737, which first flew in 1967. But that's another story.
[1] https://www.youtube.com/watch?v=vKQYG_fA2uM
[2] https://www.youtube.com/watch?v=l1-urTRxeEM
[3] https://en.wikipedia.org/wiki/Coffin_corner_%28aerodynamics%...
Using the language of the article, I’d say “push all your innovation tokens into agents” is probably a good move. This means the tech your agents work with should all be boring tech.
Another way of saying this is “use in-distribution technology”. If agents are substantially better at Rust than Zig, probably you should use Rust, even if Zig is “better”. The amount that Zig is better is going to get swamped by the amount that in-distribution agents are better.
(This is not a claim that Rust actually is better, just a hypothetical fact pattern for discussion.)
1. I once worked for a company that had a large Cassandra cluster that was primarily serving the role of a distributed append-only log. This role is as perfect a fit for Cassandra as I can imagine. When we needed a distributed database for authentication, we decided to use Cassandra for it since we had in-house expertise (it was "boring technology") and a cluster we could piggy-back on for a while. When we needed a distributed database for a key-value service, we used the same reasoning and chose Cassandra again. There were many problems with Cassandra that I won't go into, but I will say that straying too far from Cassandra's comfort zone stretched it to a breaking point. And we had some downtime and rough nights because of it. Sometimes boring technology isn't enough, you also need boring workloads for it.
2. I was an early adopter for Rust. Before 1.0, it wasn't clear to most people whether the language would mount to anything. To me, I saw a formal proof assistant being put into programmer's hands and could tell right away it had a bright future. Rust was not a boring technology back then. You might say it is now (for some use cases). Adopting shiny new technologies that measurably improve confidence is not a risk. What's foolish is getting comfortable and complacent with familiar, aka boring, technologies that are difficult to use correctly.
So many jobs are sold as “we’re pragmatist's” and when you show up there’s 5 devs, 50 repos and most of the work is discussing if x requirement should be a new micro service. The product is usually an web app with 10 entities and and an API.
Suppose it’s keeping people in jobs
It was fairly obvious it wasn't going to deliver, but usually newer and more inexperienced engineers were extremely enthusiastic and would build the ultimate mess. This was a refreshing blog and presentation that provided a way to communicate to people boring is better, and to ship product.
I think it's pretty likely that in, let's just go for the round number and say 80% of cases, everyone will conclude that just existing well used frameworks / code generator templates where everything's understood and most of the hot paths are well tested is pretty fast to build with (and more importantly, maintain and operate) and was really all we needed for most useful business software.
We're "scalable" over here. It's a sexy problem to have.
It's a case by case basis. I wonder why people try to encapsulate these things into "philosophies" that are simple minded and stereotypical.
Choose what's better. That's not always necessarily boring technology. Sometimes it is... sometimes it isn't.
At my current job we generally use LINQpad for scripting, and have a lot of tests that use LINQPad scripts.
When I joined, there was a test that relied on an "R" script. Now, "R" might be the best tool for the job, but no one running the test would ever have any "R" experience. In addition, the environment to run "R" has quite the learning curve.
I ported the script to LINQPad. The fact that no one needs to install "R" or figure out how to run "R" is a much more massive timesaver than the fact that "R" was technically better for that one task. (And the script itself is pretty straightforward.)
The tricky bit I think though, is deciding what actually is boring, since really we should factor complexity in too.
I work a lot with data in python, for in memory data, the most popular choices are:
- pandas - pyspark - polars
Pandas is the most established, although has a lot of technical limitations, as well as complexities.
Pyspark is definitely an "industry standard" choice, but now you're dealing with distributed computing when you probably didn't need too.
Polars is the simplest in terms of API and not being distributed, but then is the newest.
I can easily imagine a discussion where three engineers all agree that they should choose the "most boring" tech, but all choose a different option.
We need to remember that the purpose of using tools is to solve specific problems and achieve goals. Having no tools, too few tools, or too many tools can all hinder the achievement of those goals. We only need to select a few useful, universal, and widely applicable tools. Unless there are necessary and sufficient reasons, we shouldn't easily switch to new tools.
For the tools we choose, one must become truly familiar with and proficient in their use, continuously customize, modify, and improve them, and strive to use them to the fullest extent, thereby significantly improving efficiency and productivity, and solving practical problems and achieving goals faster and better.
The example they gave was something like they wanted to use rabbitmq for a new side project (I might be misremembering) but they were forced to make it work with redis instead. The author said that years later he found out that that side project had exploded in popularity and it had coped with it fine because the infra team were already handling the stack and it wasn't some snowflake deployment.
Does anyone remember the post I'm talking about?
Instead I like “only work on impossible problems”
Most of them turn out to be impossible, but some of them turn out to be possible.
I’ve never met anyone who could pick 3 and be confident in getting even one right. Tokens are a terrible analogy for innovation or research.
In hindsight I’ve had to sift through hundreds or more to fine one that worked.
I thought this post was helpful when I first started thinking about startups.
After more time, I think boring tech isn’t worth thinking about.
Don't let one affect the other.
Sadly, that is real out-of-the-box thinking.
And generalizations in software development are always wrong.
This should not be new.
Make your shit work. Make it work well. Don't fuck with a thing that's working as desired.
In this way, you'll make everyone's lives at least a little easier.
This should be a mandatory teaching to anyone who wants to use the moniker of "engineer".