back
260 comments
Why even have a blog when you can't be arsed to write the posts. This is so obviously LLM-written. I have a positive view of Shopify engineers, but this kind of made a dent in that confidence.
What exactly makes it "obvious" that this is written by AI? I could totally believe that AI was used to generate parts of it, but I really don't get the sense that the whole thing was written that way. I've seen way worse examples on this site.

As software engineers we are constantly told that we need to heavily use these tools for our daily work. So is it surprising that software engineers use the same tools as writing aids? Using AI does not mean no human effort was involved.

I too have a positive view of Shopify engineers. But no longer Shopify management. It went from bottom up trust basted to top down AI first and for everything based. It’s very hard to express how deeply and completely the internal workings and culture of the company has changed. (Source: I am a long time former developer there.)
I have a gemini gem that recompresses the expansion that AI causes. It seems to have worked great on this article!

https://share.gemini.google/WxEi6Satj7mY

Happy to share the gem with anybody who's interested in reading articles like this without having to muddle through all the AI bulk.

I agree, I too found this barely readable.

Interestingly, ChatGPT was able to parse and explain it to me. I got a lot out of its presentation.

I agree with you, it doesn't take long to identify the itemized style that LLMs use and it's always incredibly annoying.

I would rather read the original thoughts of the engineer, grammar mistakes and stylistic imperfections included, than prompt generated AI slop.

I think they're just following the orders of the CEO.
unfortunately its new normal now
> This is so obviously LLM-written.

Ok? Who cares. This was very interesting to me. IDGAF if AI was used (or not) to put the words in the article, the bits of information in those words is why I'm reading it. It's almost 2027, people need to accept that articles, art, videogames, code, everything will be more and more "AI touched" and you can't stop it.

I mean I share all the people's worry about AI taking jobs and a handful of AI (and AI adjacent) corps sucking out massive amounts of resources, but reading under every 2nd or 3rd article how it's AI is beginning to be more annoying than the AI writing itself.

It's like now every interesting or weird or exceptional or controversional video of anything has (what seem to be) 80-100 IQ people spamming "AI" in the comments instead of making interesting observations, posting their own anecdotes, cracking jokes etc.

I just wanted to drop a note that I almost never go into an article or blog wondering if its AI or not. I am trying to be a more picky reader and try to actually do something occasionally but I do browse database articles and this is a good one. So I wonder, am I becoming insensitive about AI writing? Am I being hypnotized into taking whatever color pill thats had a color representing a surrender to the AI "hive mind" whatever that is?

So I'm sort of curious, did you not like the AI writing style or just rejecting AI in general? I myself am very conflicted, I think AI in its current form is the wrong tech at the wrong time yet I don't mind reading AI text and sort of mooch off of googles free tier.

Also I'm starting to notice lots of AI smearing, I read folks online describing someone elses contribution as "obviously AI" and I'm suspecting in some cases these could be false accusations.

I'm thinking about starting a blog, so when my AI gf writes posts, do I ask her to try to "not read like an AI"? Anybody try that? I'm gonna try that. Hehe for all you know I already did hehe

> Instead of one row per item with a quantity column, we use one row per sellable unit. An item with 10 units has 10 rows.

> But one row per unit for all inventory would break down at scale—an item with 50,000 units across 10 locations would mean 500,000 rows, and the reserve query would slow as it scans through them. Instead, we maintain a bounded pool of available rows, capped at 1,000 per item/location combination. Reservations consume rows from this pool; a replenishment process refills it from the inventory ledger.

Shouldn't I feel uncomfortable with such approach? It seems to create a backoff (pool) for lowering the chance of having a synchronization issue.

I agree, it does seem awfully complicated and there are quite a few pieces missing for this to be a complete solution.

I'm a bit surprised about the scalability case against a simpler solution. This is not about Shopify's scale. We're talking about contention for a specific SKU of a specific seller at a specific warehouse location.

How many shopping carts are competing for a single SKU at the payment stage at peak hours? Can this really be too much lock contention for a single database row?

I realise Shopify engineers are neither stupid nor inexperienced. Hence my surprise. I would have liked to hear more about that specific problem.

Comes down to type of items, when you have physical inventory the number is limited so more manageable and interestingly enough the problem only applies to physical inventory.

You are just spending some more disk space to avoid synchronization issues. Denormalization for performance is a really common pattern, just that people do not start with it in the first place itself

I would call this one-row-per-contract-type, and this is the most general model for the problem (e.g. the model cannot be further broken down into finer level), thus, the most scalable model given storage is dirt cheap.
Mostly unrelated but shopify is incredibly annoying. They introduced this delivery tracking app called "shop" and it has become unavoidable when buying electronics from china. Recently looked at it with mitmproxy and it ships home more than gets shipped to me.
It seems there could be a simpler solution.

1. Deduct the reservation from the inventory when the user starts to order, but in the same txn also maintain a separate row for the in progress order flow. 2. If the order flow is aborted or times out have a background process that returns these to the inventory.

That seems simpler than this approach and involves no locking. Though their presented approach is also reasonable, there must be some reason not to choose a simpler flow. It is not that difficult to have a gc service that scales, but may be they didn't want to separate that.

My main takeaway from this post is that in 2026 we haven't developed enough technology to scalably and durably handle concurrently decrementing a single number. This has caused multiple organizations to develop database hacks (the multiple rows) or complex architectural solutions (redis) which destroy the atomicity of the process.
Is it really the right choice to drop Redis and go back to a disk based relational database just to wrap transactions into a single unit?

Redis handles tens of thousands of concurrent connections in a single event loop, while MySQL uses one thread per connection. No matter how I look at it, that seems like a step backward.

Of course, performance isn't everything. And if performance isn't a problem, having everything in one place does make it easier to reason about. But I'm worried that under spike traffic, this approach might actually cause more problems.

I think putting a scheduling layer in front of the DB would be a better approach. The application server could handle concurrent connections and only write to MySQL when correctness is actually needed. That seems like a cheaper way to do it. but is it different for large-scale enterprise distributed systems?

not the best design to have 1000 rows for each shop*SKU combination. If a candidate proposed this solution during Shopify's System Design interview, i doubt he would be vetted for Senior+ position.

Instead of having 1000 rows per shop*SKU, why not just have one row per shopping cart*SKU?

That way a single row would represent a single cart, and will hold info of multiple items of the same SKU.

No need a cludge with 1000 rows limit and replenishment process. Instead of dealing with N rows, you always deal with a single row.

Why is it so hard for many people to accept, that this is a solution for a specific problem of shopify? They did not say that Redis is bad and MySql is good. They only a solve their problem.
I wonder how we could handle that in a simpler way with durable workflows (e.g. Temporal, Restante, DBOS) – which are similar to Erlang processes but with persistent disk storage. This could avoid the need to maintain the 1000 row inventory.

Perhaps each shopping cart would have its own workflow, and the inventory item would have one as well. Then, whenever a customer put an item in their cart, their cart workflow would send a signal to the inventory item workflow and wait for the response. The inventory item workflow would maintain a ledger controlling to which cart each unit goes, and it could batch the writes to this table. This way, even if 100k customers try to purchase the same item in the same second, it should handle the load.

After the batch is written to the ledger, the inventory item workflow would reply signals to each cart workflow confirming that the reservation was completed. The end-to-end latency from the consumer point of view would be a fraction of a second, without needing the 1000-row hot-inventory heuristic.

"But the hardest lesson wasn't about database design. It was discovering that the real bottleneck wasn’t what we were observing and measuring."
Could not they shard the inventory table by shop_id? As I understand, the order includes only items from one store, so there is no need to keep all the stores in a single table.

Also, I wonder why they could not have a row status (available/reserved) and UPDATE it instead of deleting the rows.

Makes sense... if you are counting something in MySQL and now your counter is in Redis that's already strange

But I guess the point is that even in the MySQL scenario the 'reserved_quantities' is almost like a temporary table so either way is not the 'Real' inventory

I never spent much time with the whole NoSQL movement, it always seemed something out of people that don't get how to optimise SQL queries, or suffer from SQL allergy, only to reinvent it badly in custom languages.
This is absolutely fascinating. I enjoy real life stories like this. I went to a Node meetup in 2013 when Target had just switched to Node from PHP and it was a similar experience to see their metrics and hear their strategy.
At that revenue, why not make your own filesystem, database and index structure? There is no way mysql is the best possible software for this use case. Why stop innovation and hand everything over to ops?
I had a client and they weren't to bothered if they sold the last item twice, they would call the customer, apologise, and offer a discount on an alternative and keep the sale.
Tobi Lütke also made some rather controversial statements lately, too, agreeing with a retired TD Bank CEO that more votes should be given to the rich. On the surface, this sounds awful, but what Eric Thor actually said was that the number of votes should be tied to the amount of income tax a person pays. Considering that (from what I've heard) billionaires pay no income taxes, I'd say it's not a bad idea. No tax: no vote.
I think this could have used tigerbeetle instead?
I don’t get Why AI written technical article is a big thing here. Did we care if an article is typed or handwritten at a point?
Pretty interesting read. One thing I’m curious about is the DB size trade off. Going from a quantity in Redis to one row per reservable unit seems like it could create a lot more rows, even with the 1,000 row cap per item/location.
Shopify’s founder and their coo both fund far-right extremism, and its founder thinks only rich people should be able to vote. But anyway, they switched databases.

https://www.techwontsave.us/episode/340_shopifys_leaders_are...

The high contrast dark theme made my eyes squint and I started getting a headache within 60 seconds of trying to read the page. The war on light themes needs to end.
It’s fascinating that in order to do this, they had to remove 50% of reads and 33% of transactions from the main DB.
> 3. Consistent lock ordering: avoiding deadlocks

This section is badly written. For example, it refers to different table names than those previously introduced.

The slop shows. While I appreciate the post, I wonder why they didn't bother using an LLM in a way that would at least ensure internal consistency.

outside the slop, i liked this post that was linked on innodb locking: https://jahfer.com/posts/innodb-locks/
They are hiring with AI slop:

https://www.shopify.com/careers/disciplines/engineering-data

Pair programming and forced AI, that sounds like absolute hell. Glorification of Lütke who didn't do that much in open source and now props up his ego by thinking "AI can do it so it wasn't all that difficult all along."

I don't think he ever worked on complex parts of Ruby. The people he now oppresses did.

Ruby should note that this company is actively repelling people from using the language. I really want to switch, but then I see Claude contributions in Ruby core, the influence of this slop company, and think it isn't worth it.

Oh, and they bought DHH in 2024 for his 180° turnaround on AI. He is now an AI booster, so Rails is out of the question as well.

They were really so proud of that AI image that they just had to tack it on at the end? Did nothing but make the blog post feel like cheap mass produced slop
so this is interesting to me, im in retail i work closely with platforms ive used shopify ive used magento ive used smaller players ive helped implement various pieces of all of them.

and i was excited to get some insight, then i realized that this whole thing was written by AI and im going to guess the idea and implementation were probably very AI driven.

> The solution: SKIP LOCKED > Core idea: one row per unit, bounded by design

cool, thanks claude.

Now I'm wondering what the engineering culture is even like at shopify.

Here's the thing. I like databases, I think there's a lot of shit in this space that went and smoked a shit ton their own good stuff to come up with these pure event driven designs that lock you into event workflows with no isolation and remove the ability to do broader bulk-functions.. and then do something even stupider and say "all you need for the interface is graphql" and such service/platform doesn't give you any other way to reconcile or do reporting for your org you have to warehouse from graphql.. this is crap. So seeing a headline where shopify says they want to kinda get behind a unified database strat behind the scenes even if it's not necessarily customer facing, like that's good imo. SQL is many decades of relational algebra that makes insane computations acrossed vast sets of data pure magic and one of the best query dml interfaces of all time.

..however i dont even agree with the claim their making here that redis isnt the tech for a reservation system. redis when used correctly feels like an insanely awesome way to do a reservation system, i lurv redis for stuff like that.

I'm just gonna go forward with the assumption that current and future shopify updates are pure vibeslop. I already hate their data interfaces, but compared to other saas offerings i appreciate that they do have bulk-features.

The lengths companies will go to avoid running different pieces of software...