back
99 comments
Have nothing but praise for FoundationDB. It has been by far the most rock solid distributed database I have ever had the pleasure of using. I used to manage HBase clusters, and the fact that I have never once had to worry about manually splitting "regions" is such a boon for administration...let alone JVM GC tuning.

We run several FDB clusters using 3-DC replication and have never once lost data. I remember when we wanted to replace all of the FDB hardware (one cluster) in AWS, and so we just doubled the cluster size, waited for data shuffling to calm down, and just started axing the original hardware. We did this all while performing over 100K production TPS.

One thing that makes the above seamless for all existing connections is that clients automatically update their "cluster file" in the event that new coordinators join or are reassigned. That alone is amazing...as you don't have to track down every single client and change / re-roll with new connection parameters.

Anyway, I talk this database up every chance I get. Keep up the awesome work.

- A very happy user.

How would it compare to say something like hosted Redis, or if you wanna be more fancy ElasticSearch. I have been looking into FDB for pretty long time and have been looking for a perfect opportunity to use it. Would be helpful if you can describe your usage scenario (kind of data you are storing).
Here's one of my favorite articles on FoundationDB, where it (FDB) passes Jepsen first try: https://web.archive.org/web/20150312112556/http://blog.found...

> I ran FoundationDB Key-Value Store through every nemesis in Jepsen - including those that found failures in other databases - and FoundationDB passed all of them with flying colors.

FoundationDB is one of the coolest pieces of technology I've used in the past decade. The tuple keyspace is incredibly useful, so are the multi-key transactions. I've physically killed the power on an FDB node and FDB cluster; multiple times (heh, home servers)... and every time the cluster or node just comes back.

That's great that you are doing your own resiliency testing.

Having someone other than those officially on the Jepsen project run the Jepsen test is a good start. However, many databases have claimed to run the Jepsen tests themselves and pass, but when there is an actual paid engagement for a distributed database there are always issues that are found. That's generally true even for unpaid official runs as well although Zookeeper did pass existing tests. Every database is different and the paid engagement will design specific tests designed to break the database in question.

This was the FDB team’s stock demo in the early days. It’s a killer move.
Two quotes from the paper that I think will motivate people to read it:

"Rigorous correctness testing via simulation makes FDB extremely reliable. In the past several years, CloudKit [59] has deployed FDB for more than 0.5M disk years without a single data corruption event. Additionally, we constantly perform data consistency checks by comparing replicas of data records and making sure they are the same. To this date, no inconsistent data replicas have ever been found in our production clusters."

"For example, early versions of FDB depended on Apache Zookeeper for coordination, which was deleted after real-world fault injection found two independent bugs in Zookeeper (circa 2010) and was replaced by a de novo Paxos implementation written in Flow. No production bugs have ever been reported since."

Ehhhh, doesn't align with my experience. I think FDB is actually really poorly tested. When I was evaluating it for replacement of the metadata key-value store at a major, public web services company we found that injecting faults into virtual NVMe devices on individual replicas would cause corrupt results returned to clients. We also found that it would just crash-loop on Linux systems with huge pages, because although someone from the project had written a huge-page-aware C++ allocator "for performance", evidently nobody had ever actually tried to use it, including the author.

It's also really, really weird that their non-scalable architecture hits a brick wall at 25 machines. Ignoring the correctness flaws, it only works if you can either design around that limit by sharding, and never off cross-shard transactions, or if you can assure yourself that your use case will never outgrow half a rack of equipment.

Thanks for the quotes, I've been wanting to read this paper for some time. Great to see they went through the consensus literature and made a decision to go with Active Disk Paxos, instead of stopping short and not fully understanding the consensus they're building on. The consensus and replication protocol is such a huge part of building a distributed database.
> de novo Paxos implementation written in Flow

That's... brave. Flow is a DSL built on top of C++?

What is the Flow referred to here?
Only great things to say about FoundationDB. We've been using it for about a year now. Got a tiny, live cluster of 35+ commodity machines (started with 3 a year ago), about 5TB capacity and growing. Been removing and adding servers (on live cluster) without a glitch. We've got another 100TB cluster in testing now. Of all the things, we're actually using it as a distributed file system.

We've tried Ceph, GlusterFS, HDFS, MinIO and some others, and eventually decided on a custom FDB solution. It's a breeze to setup, and seems to eclipse others in performance [0] and reliability - Kyle (aphyr) the author of Jepsen series on distributed systems correctness, said: "haven't tested foundation in part because their testing appears to be waaaay more rigorous than mine." [1]

The way we use FDB, if anyone is interested, is we simply split files into small chunks (per FDB data design recommendations), and store all file's & folder's meta data in FDB such as byte count, create/access/write times, permissions, and a lot more. Folders are handled by the builtin Directory layer [2].

[0] https://apple.github.io/foundationdb/performance.html

[1] https://web.archive.org/web/20150312112552/http://blog.found...

[2] https://forums.foundationdb.org/t/whats-the-purpose-of-the-d...

Interesting! We have been doing the same thing with HopsFS for a couple of years. Except, we only store the small files in our database (www.rondb.com) - RonDB is a recent fork of MySQL Cluster (NDBCluster). Very small files (<1KB) are stored in memory in RonDB, small files (typically <128KB) are stored in NVMe disks in RonDB, and other files in HopsFS (which now stores its blocks in object storage (S3, ABS).

We had a paper on it as ACM Middleware and it's open-source on github. Are you going to publish your solution?

(Discussed here on HN: https://news.ycombinator.com/item?id=25149154 )

That's a really interesting solution. Can you tell us more about it? Operating distributed blob storage systems is kind of fragile with every software i have yet tried.
Markus Pilman from Snowflake did an awesome talk on FoundationDB's testing at CMU's Quarantine Tech Talks (2020), How I Learned to Stop Worrying and Trust the Database:

https://www.youtube.com/watch?v=OJb8A6h9jQQ

Here's another excellent talk at Strangeloop on FoundationDB's simulation testing by Will Wilson in 2014: https://www.youtube.com/watch?v=4fFDFbi3toc
FDB is an awesome and unique piece of software (I attribute quite a bit of Snowflake's success to FDB). I've also had the pleasure of meeting some folks from the original team and they are true engineers. Does anyone know if/when Redwood (the new storage engine) has landed / will land?
Founders are building a distributed systems simulation product now called Antithesis. My data fabric startup, Stardog, is a happy Antithesis early adopter customer. It’s helping us reproduce and fix non-deterministic bugs deterministically. Good stuff.
> I attribute quite a bit of Snowflake's success to FDB

How so?

CouchDB 4 is built upon Foundation FWIW
Didn’t know, very happy to hear
I just implemented a database with changefeeds using FoundationDB (in Clojure), to eventually replace RethinkDB in my system. Very impressed so far.
That’s awesome! I’m interested specifically in using FDB with Clojure.

Did you look at Crux as well? (DB written in Clojure, has primitives to build changefeeds, opencrux.com).

Personally I don't understand how you can call a database robust if it can't scale down nodes after you scaled them up once. What am I supposed to do if I ever deploy to 50 nodes and then it turns out that I only need 5. Shut the business down? Pay to run database servers forever that I don't even need anymore? Also the database configuration has a lot of gotchas and is very opaque. You might be waiting for 30sec for your CLI to connect to your localhost cluster of two processes and you have no idea what is happening or why it is taking that long. It just never felt so safe and robust as people claim it to be. I don't know, these were just my findings on the brief tests I did with it.

Also you better get familiar with a whole bunch of hidden "knobs" that are apparently configurable and very important somewhere and then get printed out into xml logs but of course there is no log viewer so you have to write your own. Maybe this isn't a problem for large companies but I'm providing feedback as a single user here.

I also don't understand how people can praise the c++ DSL. They should rewrite that into standard c++ coroutines as soon as possible so their entire build and dev environment isn't so hard to understand. As a user of open source software I generally like to be able to debug through the projects I use and figure out problems I have. It's much harder when a project uses their own custom language. I certainly tried to set it all up correctly but there always seemed to be some problems in regards to Intellisense within the IDE.

https://apple.github.io/foundationdb/administration.html#rem...

What is this process describing, and how does it differ from what you were trying to do?

I am pretty sure that the new cloudant transaction/storage engine is also based on foundationDB, which powers a lot of things behind the scenes at ibm. And couchdb 4 with foundationDB storage engine is hopefully not too far out either. Lets see how long this whole transition takes, but i am still hopeful that the mindshare and motivation of apple, snowflake, ibm and apache community will lead to something great.
This is a really good document:

https://apple.github.io/foundationdb/data-modeling.html

I have been studying these key-value stores with efficient range iteration lately (such as LevelDB, RocksDB, BigTable, FoundationDB, etc). This is a great reference on how to make such a simple abstraction do a lot of useful things.

Did they ever implement a SQL layer? They seemed like one of the only NoSQL products with the architecture to make it plausible to do so.
There was an SQL layer but performance was sub-par IIRC. There was also a blog post somewhere explaining why it's probably not a good idea to build an SQL layer on top of a KV store, devil in details, etc.

Edit: found it https://www.voltdb.com/blog/2015/04/foundationdbs-lesson-fas...

Not sure what to think of it, I'm not a DB expert by any means but the post sounds plausible enough and the SQL layer is discontinued AFAIK. I guess with each new abstraction layer you leave some perf on the table.

iirc, they did. But it wasn't opensourced.
They got acquihired by apple, didn't they? Was. Fdb ever oss'd?

Is it CP or AP? Comments seem to imply AP

It wasn’t an acquihire. Apple paid a lot of $$ for FDB.
This seems like a good place to ask - are there any new and exiting FOSS "application" worth checking out? I recall from the initial publication of the source - there was references to a great sql layer? I don't know if a FOSS work-a-like ever materialized? Other things I'd hoped for was a network filesystem/blob layer, like maybe s3/nfs/webdavfs compatible? What are people building on top of foundationdb today?

Ed: i suppose various document/db applications - like IMAP might be a good fit too?

large unstructured blobs and large files are among the things not well suited to foundationdb and couchdb 4 actually reduced supported blob size in the transition to foundationdb. it looks like object/blob storage systems are at the moment rather seperating more from key/value and document storage than growing together. but this is a good thing because the tradeoffs are very different and it allows each system to focus on what it does best. blob stores will hopefully move even more to content addressing and merkle dag similar to git and ipfs.
I’m curious about this as well. Is anyone working on building text search on top of FDB? It’s kind of astounding to me that last time I checked Elasticsearch was still essentially the only game in town.
peruse the fdb forum. they produce document and record layers now. there are community layers of varying quality for a network block device, a filesystem, and a few other things.
Am I right that this is like a distributed form of something like LevelDB or RocksDB, which would be the underlying storage engine for a full database product?

And/or would it be comparable to DynamoDB?

>> In its newest release, CouchDB [2] (arguably the firstNoSQL system) is being re-built as a layer on top of FoundationDB.

That is impressive. Like a framework for implementing noSQL DBs.

This is similar to WiredTiger, and its role in MongoDB.
Does anyone know of the"sqlite connector" mentioned in this post ?

https://opensourceconnections.com/blog/2013/05/06/does-found...

It would be really cool to find it, if it's still out there.

It's unfortunate that they went silent for years after the Apple acquisition. That period was key for database adoption. I have the feeling everybody kind of settled for pgsql.
> I have the feeling everybody kind of settled for pgsql.

That's probably because of spending time on this echo chamber.

In reality everyone has likely been staying with the same databases they know and love but just moved to the cloud. It's why now AWS for example offers such a wide variety of databases e.g. MySQL, PostgreSQL, SQL Server, Oracle, MongoDB, Cassandra, Redis.

Those are two completely non overlapping use cases. If you can use pgsql for your problem, you have no business trying to use a distributed key value store instead. That would be at least as dumb as driving screws with a hammer.
With little or no admin and monitoring tools.
What is the backup / restore story in FoundationDB? How does it compare to postgresql?
Didn't couchbase move to FDB for their underlying engine?
I’d love to see a good primer on data models and scenarios that are well suited to FDB.