▲ 346 points
back
75 comments
Just finished installing it on my OpenIndiana NAS to replace Minio.
Biggest difference so far is that Minio is just files on disk, Garage chunks all files and has a metadata db.
Minios listing operations were horribly slow, still have to see if Garage resolves that.
> Biggest difference so far is that Minio is just files on disk
Minio _was_ just files on disk. They don't support that mode anymore since 2022-10-29 (see the big yellow warning box at [1]).
[1] https://min.io/docs/minio/linux/operations/install-deploy-ma...
Yeah, I have actually frozen Minio at this older version in my stack, as "just files on disk" was the primary feature that drew me to it. I don't want my data locked into some custom format. I'd be willing to bet that ZFS will still be supported in 20 years, but I would not make the same bet about Minio.
For the same reason, it looks like Garage is not an option for my use case.
Ah interesting. I found it appealing to always have a way to get at the data natively as a worst case for restores. The whole use case is for Vertical Backup (from the maker of Duplicacy) to back up VMs.
> Biggest difference so far is that Minio is just files on disk, Garage chunks all files and has a metadata db.
I'd kind of expect most blob storage solutions to use abstractions other than just the file system, or at least consider doing so.
I recently built a system to handle millions of documents as a proof of concept and when I was testing it with 10 million files, the server ran out of inodes, before I went over to storing the blobs in some attached storage that had XFS: https://blog.kronis.dev/tutorials/3-4-pidgeot-a-system-for-m...
With abstracted storage (say, files bunches up into X MB large containers or chunked into such when too large, with something else to keep track of what is where) that wouldn't be such an issue, though you might end up with other issues along the way.
It's curious that we don't advocate for storing blobs in relational databases anymore, even though I can also understand the reasoning (or at least why having a separate DB for your system data and your blob data would be a good idea, for backups/test data/deciding where to host what and so on).
> I'd kind of expect most blob storage solutions to use abstractions other than just the file system, or at least consider doing so.
Honestly, I'd expect the exact opposite. Filesystems are really good at storing files. Why not leverage all that work?
> I recently built a system to handle millions of documents as a proof of concept and when I was testing it with 10 million files, the server ran out of inodes, before I went over to storing the blobs in some attached storage that had XFS
That's a misconfiguration issue though, not a reason to not store blobs as files on disk. Ext4 can handle 2^32 files. ZFS can handle 2^128(?).
> With abstracted storage (say, files bunches up into X MB large containers or chunked into such when too large, with something else to keep track of what is where) that wouldn't be such an issue, though you might end up with other issues along the way.
A few issues that come to mind for me:
* This requires tuning to actually reduce the number of inodes of used for certain datasets. E.g., if I'm storing large media files, that chunking would _increase_ the number of files on disk, not reduce it. At which point, if inode limits are the issue, we're just making it worse.
* It adds additional complexity. Now you need to account for these chunks, and, if you care about the data, check it periodically.
* You need specific tooling to work with it. Files on a filesystem are.. files on a filesystem. Easy to backup, easy to view. Arbitrary chunking and such requires tooling to perform operations on it. Tooling that may break, or have the wrong versions, or.. etc.
> It's curious that we don't advocate for storing blobs in relational databases anymore, even though I can also understand the reasoning
In my experience, the popular RDBMS out there just aren't good at it. With the way locking semantics and their transaction queueing works, storing and retrieving lots of blobs just isn't performant. You can get away with it for a long time though, and it can be pretty nice when you can.
> It's curious that we don't advocate for storing blobs in relational databases anymore
That's exactly what I did recently on new work: migrated blobs from DB to S3. It significantly reduced load from the servers (and will reduce more, right now the implementation is primitive - just proxying S3, using URL will allow other services to deal with S3 directly). It solved backup nightmare (those people couldn't do backup because their server run out of space every month). I'll admit that backup issue is more like admin incompetence but I work with what I get. Having database shrink from 200GB to 80MB now allows to backup/restore it in seconds rather than hours.
I didn't find any issues with S3 approach. Even transactions solved by a tiny possibility of leaving junk in S3 which is a non-issue. Just upload all data to S3 before commit and delete if commit fails (and if commit fails and delete fails, so be it).
I thought the "stat" command of Minio was supposed to resolve the "listing is horribly slow" issue?
Maybe, but that didn't help third party tools that I could see.
A very good alternative is seaweedfs https://github.com/chrislusf/seaweedfs/ based on facebook haystack paper (efficient small files) & more.
Did you pull that URL out of some blog post or something? The accurate URL is https://github.com/seaweedfs/seaweedfs
He (the developer) had the repo on his own profile. Then he moved the repo to seaweedfs organization. My browser auto-completed the old url.
It looks similar to minio, which as also an AGPL single binary that implements the S3 API. However Minio is written in Go and Garage is in Rust. I'd love to see a detailed comparison.
Garage design goals and non-goals: https://garagehq.deuxfleurs.fr/documentation/design/goals/
Seaweed design goals / features: https://github.com/seaweedfs/seaweedfs
MinIO: https://min.io/docs/minio/linux/operations/concepts.html
Note Garage list of non goals specify priority features of Seaweed or MinIO, for example erasure coding.
Does it have API parity with S3 ? On top of normal get/put S3 offers a retention policies and abilities to apply say a given header to a file or whole bucket
Reading that only makes Minio and Garage seem more similar. They are both single-binary, clustered, S3-compatible file storage solutions. They are both meant for self-hosting on consumer-grade hard drives, provide redundancy to deal with drive or node failures, and don't aim to implement the entire AWS S3 API surface.
We're running Ceph with Rook and Minio in Kubernetes. I'm evaluating Garage as an alternative for some purposes such as application instance static files for Python or Javascript applications to avoid block volumes in deployments. Garage has a simpler high availability distribution story than Minio's erasure coding striping, but it works well for applications that don't require sub-second writes. It can do high availability with only three nodes versus four for Minio, which works well for smaller setups. It can also serve static websites straight from a bucket, which would work well for static sites like those made with static site generators.
Ceph has an S3 API gateway, but it's a more wholesome solution better suited for larger setups compared to Minio and Garage.
It seems that Garage is designed to be fast when inter-node latency is high. A geodistributed "cluster," for example, would profit from using Garage instead of Minio.
Do we know the largest size of data that these object stores have handled? People seem have been moving away from HDFS, yet companies could host exabytes of data on HDFS and serve TBs of scans per second with a single team of fewer than 15 people. I was wondering how production-ready the other OSS alternatives are for such scale of data.
I remember working at a company that got started before cloud took off. They used mogile fs which I recently found at https://mogilefs.github.io/mogilefs-docs/HighLevelOverview.h... but I never hear about anyone else using it. It wasn't as stable as S3 but it was okay, I guess. Does anyone else here remember that distributed open source file system?
Yeah, I used it long time ago along with GearMan and Memcached.
Brad Fitzpatrick was (still is?) living in the future.
I used to work for an Alexa top 2k VOD website that used MogileFS, I bet they still use it.
Really clean docs page, nice design!
So is this supposed to be a simpler, less "omg you need a full time staff of 10 to manage" version of Ceph?
Doubtful, as Garage's features are a small subset of Ceph's. Ceph supports block storage, object storage (via RadosGW), and file storage (via cephfs). Ceph is highly sophisticated and very complex. It's anything but simple.
Related:
Garage, our self-hosted distributed object storage solution - https://news.ycombinator.com/item?id=30256753 - Feb 2022 (130 comments)
Is there any OSS implementation that is not AGPL?
Seaweed. Apache license.
I don’t think development is very active at all now that Basho is dead, but Riak CS is Apache licensed.
Ceph Object Gateway is LGPLv2.1
Why would that matter? Apps using its api don't need to be AGPL
First of all, it isn't clear this is the case.
But this entire comment thread on the AGPL misses the mark. It doesn't matter that the AGPL hasn't been tested in court or what fine grained distinctions you apply to the license or what the AGPL intends. No company in their right mind would risk using software licensed under the AGPL because the result of being wrong would be catastrophic. The legal advice to be skeptical of the AGPL is absolutely right. There is no conceivable reason to ever use AGPL software when you could simply license it under a commerical license or use a non-AGPL alternative.
Generally when someone licenses something under the AGPL they totally understand this and that is their intention.
We would like to self host and our legal team forbids us from using AGPL stuff.
This doesn’t seem accurate. Isn’t AGPL viral across RPC boundaries requiring open sourcing not just the service but all supporting code for that service?
Triton Manta is MPL
Kind of topic. But I’m wondering if it’s possible to get comparable pricing when self-hosting a solution similar to this in the cloud vs say AWS S3.
Yes. Specially S3 which is way overpriced. I have setup already a multi-region minio cluster at the cost of $0.0039/GB.
The only "cloud" solution that could get closer to that is Storj. [0]
Isn't storj using a cryptocurrency?
Thanks! Where are you hosting the machines? Are they on a vps somewhere, or do you own the hardware?
Comparison to Minio?
For this use-case, I like JuiceFS better.
* https://github.com/juicedata/juicefs
I am not affiliated with them, just a regular user.
Juicefs is not a distributed storage, you didn't read the post at all. It's just a filesystem layer on top of distributed storage (S3).
JuiceFS looks like something that could be used with Garage.
How does this compare to Wasabi for hot backups?
Wasabi is a service this is a filesystem