back

by levkk·2mo ago·view on hn ↗
I don't understand this obsession with SQLite for real, production apps. SQLite is an embedded database, completely unsuitable for managing concurrency. This is what database _servers_ are for, e.g., Postgres, MySQL, etc. Their entire job is to allow you to modify data from multiple processes, on different machines, at the same time.

This is a foundational principle of computer science. It seems to me that the "SQLite for everything" crowd is a little bit inexperienced.

36 comments
You seem to have a rather limited understanding of what kinds of concurrency exist and how those needs are best met. Whether something is a server or not is not very relevant to this discussion.

SQLite is an excellent production db for many real world workloads, as has been widely documented. It is very different to Postgres, so requires learning a whole new skill set.

One way to think about it is that SQLite can work well for the parts of your system where there is naturally strong partitioning.

> SQLite can work well for the parts of your system where there is naturally strong partitioning.

Or the parts of your system that don't have big data and no need for massively concurrent writes. And that's the vast majority of systems!

What additional skill set do you need to "learn" for SQLite? Copying files around?
For me, I have a use case that needs to support a few thousand users, probably a few hundred concurrently.

The combination of SQLite (libsql, a concurrent implementation of sqlite) and Rust means I can do so from a $2/m VPS and a single server instance.

Backups are done via a cron job that uploads to S3.

Does it pass the "Netflix scale" test? No

But it doesn't need to. I'm not profiting from the service and SQLite offers a path to scale if/when ready because... well it's just SQL and I can literally just swap `libsql::Connection` with `psql::Connection` in my repositories.

Plus upgrading from a $2/m VPS to a $10/month VPS quadripples the number of concurrent users I can support.

IMO, you can vertically scale extraordinary far with SQlite and an efficient server implementation.

I'd wager that 90% of forum websites, wordpress sites and online shops would be fine with SQLite.

> The combination of SQLite (libsql, a concurrent implementation of sqlite) and Rust means I can do so from a $2/m VPS and a single server instance.

You can probably do it with regular SQLite, too. Being limited to a single writer isn't as devastating as it sounds when they get processed very quickly. Probably don't need Rust either but it'll be more efficient than the usual choices.

(Also, it looks like libsql is the same as SQLite? Only Turso has concurrent writes)

Why do you need libsql? Single writer tends to scale better than concurent writes.
There are many cases where SQLite + concurrent front end (like a go net/http server) can handle all the load that a service might ever conceivably have to handle, especially if allowed to scale up hardware over time. You can trivially scale up SQLite to, what, hundreds of thousands of tps?

The only thing you really give up is HA/failover and DR. But there are solutions to deal with those. And single-server systems are generally surprisingly robust (since, in the absence of very complex control planes, uptime goes down with more systems).

Why go through the trouble of shoehorning SQLite into a cloud database by getting solutions for HA/failover and DR, when you can just use Postgres off the shelf?
You also have the issue or normal maintenance (patching, OS upgrades, etc). You can’t do those without downtime if you are using SQLite.
I was thinking of using SQLite on top of k3s/Longhorn to replicate it. Anyone do something similar? Folks mention light steam and aws but Jeff Bezos’s biceps are too much for me to handle.
That’s why there are billions of SQLite databases right?

SQLite is likely used more than all other database engines combined. Billions and billions of copies of SQLite exist in the wild. SQLite is found in:

Every Android device Every iPhone and iOS device Every Mac Every Windows 10/11 installation Every Firefox, Chrome, and Safari web browser Every instance of Skype Every instance of iTunes Every Dropbox client Every TurboTax and QuickBooks PHP and Python Most television sets and set-top cable boxes Most automotive multimedia systems Countless millions of other applications

https://sqlite.org/mostdeployed.html

That’s a comprehensive list of single user devices.
levkk is talking about concurrency. The list you gave doesn't explain high concurrency requirements for usage.
GP calls out concurrency as a weakness of SQLite. Most of the examples here don't experience the same load even a moderately sized web service experience day to day.

And no, being a part of the python standard library doesn't means it is being used by the average python user. These days I'd say at least half of them are just there for machine learning.

I mean... if you count flat files as "databases", there are a heck of a lot more
And I don't understand the obsession with server-based databases for single apps. Especially in containerised setups, every "app" gets its own database anyways, and if the app is further broken down into services, they usually communicate between each other and not with a shared database. So in those cases, what do you gain by pulling the database out of the "process" and onto the other end of a socket? In most cases, absolutely nothing. So why bother?

Don't get me wrong, I've worked with plenty of server-based databases, including proper dedicated database servers. It's great tech and often the best tool for the job. But not always and I'd argue not in the majority of uses.

Sqlite is good for lots of stuff, but you're probably focusing your days on high-scale webapps that want sharding with networked DBs. That's one domain, and an interesting one, but there are lots of others.

I'm a big fan of re-evaluating prior "best practices" in light of technology changes, especially in ways that improve simplicity. Running my family's social media site off a single sqlite DB on a VPS is great. ~15 users, almost zero maintenance. I run my FreshRSS instance off of sqlite, as well as my "now" page. At work, I used sqlite for all kinds of things over the past decades: as an ad hoc job queue, as a quick way to ingest and query lots of logs locally, and present/filter in realtime with simonw's excellent https://github.com/simonw/datasette.

I don't think it's every "sqlite for everything" as much as it is "sqlite in lots of places you probably didn't think to apply it."

kentonv/Cloudflare's work on sqlite at the edge might have made this thinking a bit more popular, but it was always around. https://blog.cloudflare.com/sqlite-in-durable-objects/

I suspect being aware of all those little neat cases and wanting to leverage sqlite for them may be an indicator of experience, rather than the opposite.

There is something appealing about "it's just a file" (it really isn't; it has locks and a WAL), but I agree with you.

I think people are afraid to read the documentation for postgres. You can start it up in milliseconds. Fast enough and light enough to run one copy for every test case in your test suite, or whatever you're using it for. (mkdir /tmp/whatever; initdb -D /tmp/whatever --no-instructions -A reject -c listen_addresses= --auth-local=trust --no-sync -c fsync=off -c unix_socket_directories=/tmp/whatever -U postgres --no-locale; postgres -D /tmp/whatever) Now you have a test database that behaves exactly like production because it's exactly like production. (OK, turning fsync off makes it a lot faster than production, so be careful.)

So teach them. If you want to bring up computer science fundamentals, the question is where does SQLite sit with regards to the CAP theorem. Consistency, Availability, and Partition tolerance. SQLite isn't a distributed system, so there are no partitions to tolerate, so it's a CA system. Other databases make different tradeoffs. For systems that don't need concurrent writes, SQLite is pretty great! There are no users to manage, no permissions, no daemon to run, no server and port to mix up. Just open a file on disk using a library.
Thing is SQLite scales better than both those network databases [1] if you're prepared to stick with one big machine (+ a standby).

This is even more obvious when you start doing transactions processing an row locks across the network limit you to 1-3k TPS that you cannot scale out of (Pareto distribution is merciless).

[1] - https://andersmurphy.com/2025/12/02/100000-tps-over-a-billio...

> SQLite for everything

is just wrong, and I don't think that the SQLite fans are that crowd. Taking a database server for everything is probably possible, but often unnecessary. With experience, one can properly judge when SQLite is sufficient and when it is not.

So arguing that the SQLite crowd is inexperienced feels weird, because inexperienced people have a much harder time judging when to use what and can just use the database server all the time (even when it is overkill).

I had very good results giving 1 SQL DB per go routine, so the accesses were serialized up front, on a very high volume (130K requests/second) service. Exact transactionality was not a product goal, and the SQLite was just to backup the in memory state. If we lost a little due to abend or something, that was ok (although for normal maintenance it caught SIGTERM and stopped the listen and then waited for in flight calls and then flushed the remaining changes to SQLite; then on startup it would read the SQLite into memory to populate before taking the listen; persistent storage across container runs, and never both reads and writes to the same file at the same time. (It also just closed the DB and opened a new one when it hit some limit of rows, so as not to fill the disk; the max size of the SQLite corresponded to the max size of the LRU map being served from in memory; then it just flipped A / B between "a full memory worth of data stored" and "the currently updating state." A lot easier than having to write out proto bufs to disk or whatever I would have done for transient (during restarts/maintenance) persistence.
Computer science no more get its hands dirty with concrete software than physics primarily being about building bridges.

It is not «a foundational principle of computer science».

> This is a foundational principle of computer science

How exactly is this a foundational principle of computer science?

I worked on an app that had sqlite databases per user... it was fine.
I think you'd be surprised to learn how many real production apps are actually running on top of SQLite (by way of Cloudflare D1).
Well if you run a tiny single-threaded app then SQLite is a nice simplification over spinning up a separate machine for Postgres.
If your data is naturally sharded (users) with writes happening within a single shard, parallelism becomes easy. The request is routed to the shard hosting the user's data and reads/writes locally.

This makes scalability _much_ easier to reason about. It's cut-paste, cut-paste. Every N users needs another shard.

It does buy you a _different_ set of problems, like cross-shard querying (analytics) and how to do load leveling as users age out.

But it avoids the whole shared index scaling problems from inserts/updates with large user counts.

It becomes a hierarchical instead of a relational database.

It's touted by the people who use the word "just" a lot.

"Just use postgres" "Just use sqlite" "Juse use a monolith" "Just use sftp" "Just use an ec2 instance"

Usually these people have flunked out of the school of (distributed system) hard knocks. They couldn't hack it and are retreating to familiar.

The funny part is when one of those people fluke themselves into senior management when their saas takes off.

Inevitably they have to suck it up and hire experts in the same technologies that "no one needs".

there is a difference between concurrency in a distributed environment and concurrency on a single machine across processes. SQLite is incredibly useful for the latter.

you seem like the inexperienced one to me..

Personally I like Postgres for this reason too. Its extremely easy to run with Docker, I can dump data from all kinds of apps in there and I know it's not going to take any rearchitecting as soon as I need multiple concurrent writers.

I think docker is still super underappreciated so setting up any kind of server is seen as a chore. In my eyes it makes running tons of services like this very easy, so ill take the extra functionality, extensibility etc of postgres.

Sure, SQLite doesn't solve every problem -- but in many cases it solves the need at hand with the reward of one less piece of infra required to support it.

I see obsessions with tooling/solutions constantly from experienced devs who fall in love with the original solution and think it's the only way to do things -- so the experience part cuts both ways.

> SQLite is an embedded database

Yes, but that's not its main selling point. An SQLite database is also a single file, which makes it incredibly easy to replicate, backup, transfer, restore, etc.

Isn't concurrency also limited by your machines disk speed for writes, what difference does it make if you write sequentially vs concurrently? Why does concurrency even matter for databases?
sqlite is more like a file format than a database. it competes with .xlsx.

> "SQLite for everything" crowd is a little bit inexperienced.

every time i see it in a real application, it becomes a huge focus of issues (for example: jellyfin, hermes, openwebui, comfyui)

Someone with experience would know that concurrency isn't a universal requirement.
It's almost as if Postgres isn't perfect, and one size shoe doesn't fit all.

Some people want some of the benefits you get from SQLite.

SQLite is obviously not perfect, but it's an incredible piece of software, and people regularly find good ways to make use of an excellent pieces of software.

I absolutely 100% do not understand it either. At all. Every time I try to over the last year or two I come away with the conclusion its something that sounds cool (to me too!) but is guaranteed to cause more problems than more obvious solutions.

That being said I'd kill for someone who used it and benefited to explain it to me in a practical sense. (specifically where syncing is involved, and syncing a subset of the SQLite is necessary. If it's "just" a document store thats treated like a blob for syncing/backup, that's familiar. If it's all in one storage but only local, that's familiar.)

Re: TFA, I guess it would have helped if I knew what Obelisk was, which is on me, and a more in-depth explanation of how this ties into AI/agents, which is on the industry/writer.

And of course there are now several responses proving your point.
I think the SQLite website itself says it best:

> SQLite does not compete with client/server databases. SQLite competes with fopen().

Most apps do not actually need the concurrency capacity that Postgres or MySQL are designed for.
I mean - I agree for the typical multi-user, SaaS webapp. But I don't think that's what these folks are proposing. If they are - yeesh, count me out.

If on the other hand they're talking about single-user, software in the small - hell yeah. In fact, I'd also promote DuckDB in this regard (mostly for analytics) - with the power of a single machine these days, you can do a surprising amount and never have to worry about distribution. Unless you know you'll have to, in which case you're probably just digging yourself a hole?

you also seem to underestimate the performance you can get on one machine.
Scale to zero is very useful.
SQLite also gets really slow at around 50 million rows.
Are you one of my enterprise customers? What if your workload does not require write concurrency?
How many production apps do you think have enough users to justify these huge DB servers?