▲ 197 points
back
77 comments
How does it compare to other players in the area? E.g Ceph, Gluster or Seaweed? (I'm no expert myself, only used those as a consumer of already setup systems)
EDIT: There is a whole comparison section in the docs that I missed: https://juicefs.com/docs/community/comparison/juicefs_vs_cep...
Bit weird comparison. Like sure CephFS doesn't support S3-like access... because the object store is a separate service that also runs on top of Ceph/RADOS store
It is weird, but it's also a valid use case. I can imagine someone wanting to pull files from a FS that was populated as a regular POSIX filesystem through an S3 api. I'm not sure if you can access the CephFS files from the underlying Ceph store easily.
It seems to be a specific part of SeaweedFS, i.e. the "filer+client" components. It will use a database or key-value store for metadata and a blob store for data, and expose that as a filesystem.
The difference is that SeaweedFS has its own blob store ("volume server") while JuiceFS uses S3 (or some other protocols). SeaweedFS also decouples the server ("filer server") from the client ("mount" command or client libraries) while JuiceFS only has a single process, so the machine where you mount the filesystem talks to the metadata and data backends directly; this means you can't mount a filesystem on an untrusted machine if I understand correctly (you need full R+W access to the backends from the machine where you mount).
You can see it as similar to `rclone mount`, which allows you to mount a remote S3 bucket locally. The difference is that JuiceFS is much faster and filesystem-like, by storing the metadata in a separate faster database and by chunking your files in the backend rather than storing files unchanged in the bucket.
What I really want is a filesystem I can span across geographically remote nodes that's transparently compatible. I should just be able to chuck files into it from my NAS like any other. I think Mayastor [1] might get some of the way there?
It compares pretty well to the proprietary WAFL/FabricPool technology by NetApp:
- have "hot" blocks on local storage (SSD cache) - "cold" blocks are stored on S3 - POSIX semantics on top
Having worked with NetApp technology for >10 years this is a welcome addition on the Open-Source side of things.
The comparisons are helpful, but I'm curious why Ozone wasn't included, since that seems like the most directly comparable alternative.
Apache Ozone is not POSIX compatible, even with the File System Optimized format [1].
https://ozone.apache.org/docs/current/feature/prefixfso.html
>JuiceFS has introduced S3 gateway since v0.11. The feature is implemented based on the MinIO S3 Gateway.
The MinIO S3 Gateway was deprecated: https://blog.min.io/deprecation-of-the-minio-gateway/
I don't know if JuiceFS is treating it like a fork that they are maintaining, or if they have other plans.
We have a fork of MinIO at https://github.com/juicedata/minio, which will be maintained by us.
Damn, I was hoping it was pre the AGPL cut-over, especially in light of juicefs's Apache 2
Adopted SeaweedFS few months back. Never looked back since then. It's fast even on HDD disks.
I love this project and I'd love to switch to it. Hopefully constructive feedback: The big issue I always run into with this stuff is what am I supposed to do if something goes wrong? I think the project documentation people would be wise to document procedures to do when certain things go wrong and how you should deal with them, such as if a server or two fail, or there's some unexpected corruption. Without that, "distributed storage" systems really feel incomplete to me. Storage is usually "mission critical" and they had a procedure for every single thing that could go wrong on the Apollo mission.
By default, it starts up exposing the file server on all interfaces, not just localhost, AKA insecure by default.
Such a poor security choice, makes me question the entire project.
Can you share your experience? What were the alternatives? Did you consider it against AWS S3?
Seaweed is excellent, use it in a bunch of places both big and small!
That looks awesome! What else did you try before going to SeaweedFS?
AWS EFS, MooseFS & Ceph.
> 99.99999999% (10 9s) reliability SLA
Can someone tell me in practical terms what that means?
1 second of unreliability every 317 years?
99.99999999% reliability means you will not loss more than one byte in every 10 GB in a year.
JuiceFS uses S3 as the underlying data storage, so S3 provides this durability SLA.
Important to note that S3 does not have any Durability SLA. We promise Durability and take it extremely seriously, but there is no SLA. Much more of an SLO
As you allude to in your response, that's usually referred to as durability, not reliability. The home page could probably use an update there to reflect that terminology.
It sounds like not very practical metrics, since losing one byte often makes whole dataset useless (encryption, checksums failures).
Do you know if strong read-after-write consistency is supported (as in s3)?
Is an atomic put-if-absent method supported in JuiceFS (as in Azure blob storage)?
If so, this could be a really cool platform for formats like Delta.io :)
It seems not, instead it provides 'close-to-open' consistency, as documented here: https://juicefs.com/docs/community/cache_management/#data-co...
JuiceFS supports create-if-not-existed by using the Java SDK (HDFS compatible), so I guess it should work well with Delta.io.
Regarding the topic of "cloud storage" - could someone tell me if Juice or maybe MinIO would be a good solution to:
1. Storing multimedia data (image/video) uploaded by an user - here I would guess it can either hit it directly or via the backend for auth
2. Should be accessible by an URL exposed outside of the docker-compose so it doesn't need to go through the backend REST API
3. Some form of authentication based on the JWT token in the Header - or maybe as this is a MVP simply generating a long enough random string will be enough
Or should I simply use nginx + filesystem and not overcomplicate?
I hear everywhere S3 but as it's a pet project don't want to go the AWS route, instead maybe a Hetzner VPS with docker-compose to run the whole setup with an external Postgres instance.
webdav, oauth2_proxy, nginx. That’s all you need. You can create your own issuer or also use dex I think.
Fancy authnz is easy to do with openresty instead of vanilla nginx.
Alternatively just use own cloud/nextcloud
I tried putting Postgres on JuiceFS and let's just say.. it didn't perform very well
Is that a use case they are really targeting though? Their splash page mentions big data with model generation and genomic sequencing as examples. I can really only speak to genetic sequencing. The IO pattern for these workflows is almost all streaming reads/writes. Random access takes too long when you are reading/writing 100-500GB files.
Postgres doesn’t like running on NFS either to be fair.
postgres + rocks fdw would be a more interesting test case (or any LSM DB)
Is it 'Posix Compatible' or 'Posix' aka 'Posix compliant'?
It's incredibly hard to make a distributed posix compatible filesystem since you run into CAP. I believe (but am not certain) you are caching locally in violation of Posix or you are signing up for arbitrarily long stalls and a ton of latency on every read/write. (I'm not certain because I'm not sure what Posix specifies wrt stale reads and other cache consistency requirements between sync's)
It would be interesting to hear what the tradeoffs are here, but assuming they are explicit and can be designed around this seems very useful.
It is not posix anything. It provides a compatibility layer that makes open, close, read, and write work but other than that does not provide the type of features that would allow you to deliver mail on it with qmail or whatever. It is incredibly misleading to advertise it that way.
As you say there is no free lunch with distributed filesystems. Application programmers have to program their way around the fact that something like posix atomic writes with multiple writers is never going to work, and that the only way to get reasonable efficiency out of the thing is to defer work until the file is closed.
Agreed, it's very hard, that's why GFS and HDFS had give up some parts of POSIX compatibility.
Per CAP, it's addressed by different meta engines (CP system, Redis, MySQL, TiKV) and also different object stores (AP system). When the meta engine is not available, the operation to JuiceFS will be blocked for a while and finally it returns EIO. When object store returns 404 (object not found), which means it's not consistent with the meta engine, it will be retried for a while, may return EIO if it's not recovered.
The file format is carefully designed to workaround the consistency issue from object store and local cache. Any part of data is written into object store and local cache with unique ID, so you will not go stale data once the metadata is correct [1].
Within a mount point, JuiceFS provides read-after-write consistency. Across clusters, JuiceFS provides open-after-close consistency, which should be enough for most of the applications, also provide good balance between consistency and performance.
[1] https://juicefs.com/docs/community/architecture/#how-juicefs...
I was actually building something similar to Juice using S3 as an object store and optionally using redis(fast) or s3(slow) for metadata storage. Basically a log structured filesystem using rolling hash chunk encoding and delegations. I kinda stopped when I found juice (and to some extent seaweed) as they were much further along. If you need shared storage and don't have crazy performance requirements it makes a lot of sense to separate out metadata and just throw blobs into object storage.
I tried to take a look into the documentation but it seems to all be in Chinese?
Edit: For some reason my phone defaulted to Chinese but on my laptop it's fine. User error I guess!
Does JuiceFS scale horizontally? I can’t see anything about how the servers federate/balance load or if they can at all.
[EDIT] looks like there's an issue -- https://github.com/juicedata/juicefs/issues/345
But this still doesn't really answer it -- if I run JuiceFS S3 Gateway in 2 places, is there any way to redirect reads?
Usually the meta engine or object storage can scale horizontally by itself, JuiceFS is middleware to talk to these two services.
To serve S3 request, you can setup multiple S3 gateway and put a load director in front of them.
Can it do full encryption from client (transfer + at-rest) with Fuse?
Currently I use ext4 image + LUKS + NBD over SSH tunel, it works but is extremely slow.
Yes, the data can be encrypted [1] by the client before sending to S3, but the metadata is not encrypted.
Is this basically a non-POSIX* FUSE for S3 and/or Redis?
* POSIX implies a whole lot of guarantees, like atomic file renames/moves, that definitely don't seem to be included here.
Atomic file/directory renames/moves is the fundamental feature of JuiceFS, which makes it truely a file system rather than a proxy to S3, please check the docs for all the compatibility details [1].
I run a mastodon instance, and can imagine hosting the storage of multiple instances on a juicefs S3 gateway... needs dedupe because I consider that
Mastodon needs S3-compatible storage. I am not sure if there's an advantage using JuiceFS to expose an S3 backend over the S3 API.
Does it support POSIX ACLs as well?
The JuiceFS Cloud supports ACL, but open source one does not support it yet.
I can't seem to find docs in a language other than Chinese on the site
The docs are in English and Chinese, there's a language selector in the top right.
Perhaps if your computer/browser's language isn't set to English it defaults to Chinese?