▲ 133 points
back
77 comments
Slightly off-topic but CedarDB is extremely exciting. It's the commercialization of the widely cited Umbra research DBMS [0] that has been in the works for several years, which benchmarks faster than DuckDB for OLAP [1] whilst simultaneously being really strong for transactional workloads. Also discussed recently here [2].
A little sad finding out it's proprietary but to be expected I suppose.
It's neat seeing postgres gearing up for async support. There's also folks like OrioleDB doing massive revamps of postgres & doing disaggregated storage in public. https://github.com/orioledb/orioledb https://hn.algolia.com/?query=orioledb&sort=byDate
Oh they were bought by Suprabase two months ago... Fingers crossed! The Suprabase CEO commented at the time, with a nice basic overview, https://news.ycombinator.com/item?id=40039138
> which benchmarks faster than DuckDB for OLAP [1]
that link doesn't do any performance comparison. They claim that CedarDB executes less "code branches" than duckdb, which may or may not translate to faster performance.
Author of that blogpost here.
> less "code branches" than duckdb, which may or may not translate to faster performance.
In that case it was about 2.5x faster than DuckDB end to end, so a bit less than the difference in branches.
If you want to see some independent benchmarks on Umbra, our underlying technology, its currently first place on Clickbench [1]. You can compare against duckdb there as well.
clickbench is a toy benchmark: small dataset, very specific queries.
Benchmarking full tcp-h (not just one query like in your post) on sizable dataset (few TBs) would be very good close to real world scenario, but vendors usually avoid this.
This "non-production mode" has a bad side effect where benchmarks on laptops will be artificially fast and people will "forget" about the data loss disclaimer, just like the bad old days of MongoDB.
The article also mentions that enterprise SSD often have supercaps & can be basically free because there's enough runtime (even if power goes out) to persist any in-flight data.
So it seems like it wouldnt be an issue. But it would be running your dev & prod in two different modes.
In Postgres, synchronous_commits = off can seem like magic during benchmarks but in real-life applications the anticipated benefits can be underwhelming. Ultimately, a reliable non-volatile write cache, sized for your workload is the answer. It evens out all the peaks and valleys of flash writes and hands control back to the database engine so it can get on with its work.
> Ultimately, a reliable non-volatile write cache, sized for your workload is the answer.
Author here, I agree! It's quite sad that we need such an involved solution to offset the inherent complexity of the flash medium (latency spikes, erase blocks, ...). We nearly had the perfect solution with Optane[1]: 100ns latency, instantly persisted writes and all that good stuff.
I'm still not over Intel killing it while I did my PhD on it.
More industry experience in real world would've made it clear that they're totally useless for datacenters where there are rarely power outages with many 9's of electrical uptime measured in months or years.
UPSes and BBWC evolved to bring reliability to production gear running in non-DC environments when mainline servers used spinning rust without backup power. Today, it's largely a vendor up-charge.
Write barriers cause far too much latency in practice on servers in tier IV datacenters, so they're almost always turned off except for a tiny fraction of systems.
There has never been a "perfect" or a universal solution, only a risk budget and suitability for a specific use-case.
There used to be battery backed ramdisks with SATA interfaces. Are they history now? These days it would be NVMe or whatever, of course.
Related:
"What Modern NVMe Storage Can Do, and How to Exploit it"[1] by Gabriel Haas and Viktor Leis.
Sometimes its the LEAST expected thing (that one this article does not cover).
Some time ago I had some really cheap SATA 2.5 SSD 256 GB in my laptop.
If generally worked well - system (FreeBSD) was responsive but ... when I deleted large file (like 1GB+ size) then the system almost FROZE ... until file is deleted.
After some investigation with Allan Jude (thanks again) we came to the conclusion that a single TRIM operation (for delete) take about 500ms (half a second) ...
By default ZFS on FreeBSD uses up to 64 TRIM operations PER SECOND (the vfs.zfs.vdev.trim_max_active setting) - so the SSD was literally DoS with TRIM operation and it crippled it to its knees.
After switching this option from 64 to 1 it solved all the problems and everything was smooth and fast again.
Details below.
% grep -A 1 -i trim /etc/sysctl.conf
# ZFS DELETE FUCKUP TRIM
# vfs.zfs.vdev.trim_max_active: 64 -> 1
vfs.zfs.vdev.trim_max_active=1
Regards,
vermaden
Maybe slight off topic, but does anyone know what the deal is with the NVMe K/V command set spec? There was a fair bit of noise made about it at the time, but I’ve not seen any drives that support it (despite a few enquiries)
Those new command sets that require a re-write of your entire software storage stack exist solely to satisfy the hyperscalers for whom re-writing the entire software storage stack can be a worthwhile optimization effort. If you don't command enough purchasing power to already be getting custom SKUs from your drive vendors, they don't have enough incentive to add those features to the models they offer you.
I imagine it could be cool for some DB vendors too, so they can optimize base on direct access and skip all OS/FS overhead.
I think the plan is for ZNS to never be available at retail and I imagine key-value will be the same.
Cool! They used my bench-fio and fio-plot tool for the 3D graph :-) [0]
What the 3D graph (also) shows is how poorly SSDs actually perform in single-threaded, low queue depth situations. Many enterprise or datacenter grade SSDs often perform much better and consistently on this front than the average consumer SSD.
How far we've come from spinning rust on dev boxes and mainline warm servers back in 2011.[0]
0. https://web.archive.org/web/20111112045055/http://buyafuckin...
People don’t warm servers before entering production anymore?
"Smaller" servers, not really much any more. With high speed networking/SSD even 64GB-12GB servers warm up really fast. Back in the disk days that could take a very long time to read from disk especially if it was random IO.
Small optanes (50-100 gigs) are quite accessible cost wise and solve most of the queue depth problem
Optane is neat tech, but it's also dead tech.
Tell me about it...
Was just too expensive compared to flash.
But the tech was definitely awesome and I hope it'll come back in one way or another.
When designing CedarDB, we recently had multiple instances where we thought: "If we had just a few KiB of Optane here ... "
>dead tech.
Agreed in general, but surely for this usage case (i.e DB) it would still solve the problem, no?
Or am i missing something here
A little off-topic, but since the article mentioned macOS not necessary actually doing a flush even on fsync(), I've been wondering how long it is before macOS actually writes to the drive. A non-trivial number of my writes are a compile followed by, oh yeah, quick fix, recompile. It'd be nice if it would hold off for a while. Given that I'm on a laptop, the risk of losing data is limited to the very rare times I actually run out of power or a kernel panic.
This seems doesn't seem right:
Unfortunately, my laptop, and probably many other developers’ laptops, do not have
an enterprise-grade SSD, or even the option to install one.
Enterprise M.2 SSDs are also a thing (Kingston DC1000B).
If they're using a mac and are limited to external connectivity, then they could throw the above into a thunderbolt enclosure.
Even when there are enterprise SSDs that are mechanically and electrically compatible with the bays/slots in a laptop, they're often untenable due to the lack of idle power management features on enterprise SSDs (because sleeping when idle kills the latency metrics). Consider that the 480GB DC1000B is officially rated for 1.9W at idle and in practice I wasn't able to get it below 1.2W. Consumer SSDs with working power management would idle under 10mW.
Thats hardly untenable. Most computers draw a lot more watts where you wouldn't miss 1.2W imo. Even these "low watt" computers are drawing like 5-7 watts at idle and once you do anything on them its a lot more than that.
Interesting, haven't seen mention of that aspect of enterprise ssds at all.
It's also a strange comment: Enterprise DBs are usually run on enterprise-grade hardware in production. To be sure, you can test them on consumer-grade hardware, but if most folks are going to run this stuff in production using enterprise-grade SSDs, the software should be optimized for that hardware. Mechanical sympathy is an important software engineering practice.
The problem of commit latency bottlenecking global throughput seems to be a matter of treating the device as a black box that must be quiesced as a unit. If the NVMe device presented multiple namespaces to the operating system, and if other unnecessary abstractions like the filesystem are discarded, then the database can make progress on one namespace while blocked on another, even within a single host.
There is a feature known as NVMe namespaces but not sure how well it's supported outside of expensive enterprise gear https://unix.stackexchange.com/questions/520231/what-are-nvm...
To a close approximation, NVMe devices that come in U.2 or U.3 form factors support multiple namespaces, and those that come in M.2 support only 1, and 4KiB LBA format support follows the same pattern with a few exceptions. But you don't have to spend an insane amount to get these features. The Micro 7x00 series costs ~$100/TB.
Isn't this basically what FUA does on SCSI? Presumably O_DIRECT+O_SYNC/O_DSYNC will invoke this path when possible?
Yeah but a hard disk drive has no internal parallelism, while an SSD has immense internal parallelism.
> If you really want durable writes with sub-millisecond latency, you might want to pay the ~20% premium that vendors charge for such enterprise SSDs.
A 20% premium for an enterprise SSD? No. A 120% premium is closer to it.
ref: https://www.newegg.com/Enterprise-SSDs/SubCategory/ID-2021
> If you want your data to be stored persistent for real, you need to issue a sync command to make the operating system block and only return control after the data has been persisted.
Well on linux the work that has to be done during a blocking sync() can be minimized by triggering async writeback via sync_file_range in advance.
> Instead of writing every single change to the SSD immediately, we can instead queue many such writes and persist the whole queue in one go. Instead of having one latency-sensitive round trip per commit, we now have one round trip per queue flush, let’s say every 100 commits. We just have to be careful not to tell the user that their data has been committed before the queue containing their data has been flushed.
I'm a little bit unclear, is the suggested method here issuing 100 `write` commands and then a `sync` (or something similar) and only informing the downstream of success when the `sync` is finish? Or is the DB actually maintaining a queue of commits to be written that get fsynced all at once?
CPU: "Hey SSD, I need to write 100MB of data."
SSD Controller: "Okay, go for it."
CPU: <Intense Writing Noises />
SSD Controller: "(Hey OS, can you dump what the CPU's throwing at me in your RAM instead? Don't tell him.)"
OS: "Okay."
<100MBs later />
CPU: "All done! You got it, SSD?"
SSD Controller: "Uhh, yeah, one sec."
SSD Controller: "(Hey OS, throw me all that data before the CPU gets wind of this.)"
OS: "Okay."
OS: <Intense Writing Noise />
SSD Controller: "Alright, come on finish writing ya damn NAND."
NAND: "Hold on, man."
CPU: "Sitrep, SSD?"
SSD Controller: "Almost done!"
SSD Controller: "All done!"
CPU: "Great!"
---
Long story short, big bursty writes are faster than small intermittent writes and SSD controllers are bloody liars.
We used a small embedded db in our desktop product only for durability and SQL semantics. After we got multiple database corruptions we switched to a small file based store written in Java with simple DB semantics.
Looking carefully at what happens when data is lost we found that it's basically all recoverable.
This lead to an amazing performance, as data is flushed only when necessary. We flush only every 10 seconds now or when the write through cache is full.
My lesson learned from that is to carefully think about the actual requirements when it comes to transactions and having loose constraints may be the quickest way to speed up performance.
For many applications even a second or two of lost data may not be a big issue.
Slow is relative. Slower than rated SSD speed I could believe.
Slower than comparable HDD? Slower enough that it impacts performance visibly for the same load?
It's been a while since I was testing but sustained FUA ( Force Unit Access or something like that), writes, which are used by some DB's in stead of cache write + sync degrade consumer SSD'S to HDD levels of performance. Perhaps not as slow as HDDs themselves would degrade under the same workload, but certainly 10s of milliseconds.
So yes, visible, extremely so. On drives commonly known as excellent consumer drives and that I myself happily use on a general desktop/gaming PC.
From the limited testing available from my personal budget it would suggest that newer drives are still impacted, although not as badly, and larger drives are ( perhaps obviously ) impacted much less than than smaller ones ( 2tb vs 256gb ). I've seen 2 level drops too.. first when cache/slc writes are overrun, and then a second, when I think the drive is forced into some kind of garbage collection/reorg at the same time.
But all that stuff is very controller specific so it's a little hard to generalize.
But equally, I now buy old enterprise drives, because they work so much better under these workloads at equivalent sizes, even if consumers drives would appear faster according to spec sheets or "drive friendly" benchmarks. I personally use old intel ( 37nn and 46nn series ) but that's mostly just what I know, there are similarly performing models from most enterprise ssd vendors
The seemingly critical parallel job performance issue sounds like it can and should be handled transparently by the OS when multiple pages are involved, and should thus be an area of research if it isn't already.
Frankly, the idea that it isn't already a solved problem makes me wonder if that could possibly be the case...
The writes thing... Outside of very rare circumstances, I would think that properly batching inserts/updates within transactions at the app level would resolve half of that, and multi-user concurrency would buy back the rest.
What am I missing?
P.S. I'm mad at the author for enlightening me to the real-world performance impact of enterprise SSD. i'm about to buy, and that might just cost me...
> The seemingly critical parallel job performance issue sounds like it can and should be handled transparently by the OS when multiple pages are involved,
A thread can only fault on one page at a time, and only sequential access patterns can benefit from the OS bringing in multiple pages per fault. Having all of your CPU cores busy handling page faults and context switches to threads that are about to trigger another page fault is only going to generate enough read traffic to keep a few SSDs busy, not a full shelf of 24+ drives. mmap() and synchronous IO don't scale well enough for today's SSDs.
That sounds like it would be an issue for anything. If the kernel knows that N pages need to be paged in, it should be able to optimize that behind the scenes in whatever way is needed.
I'll agree that mmap() would probably have issues with prediction there (other than basic readahead), but I would think that issuing a simple bulk read() call (or more likely an equivalent zero copy call; vread() if memory serves? Been too long) would give the kernel enough information to optimize bulk transfers.
Small async reads will always be slow if done serially, but there's already "standard" scatter/gather calls that should let the kernel optimize that as well.
In short, all but a truly naive application should be limited by the SSD or PCIe bandwidth before they ever hit issues like that if the kernel is optimizing correctly, and assuming that the application can predict its own needs.
...when they run out of SLC cache. Fortunately it looks like enterprise SLC SSDs have come back into popularity somewhat, and for those who don't want to overpay, some cheap TLC and QLC drives can be forced to run in full-SLC mode too: