▲ 19 points

Show HN: Version code, models, & datasets together in GitHub

by skadamat·2y ago·6 comments·view on hn ↗

Hi HN! We just launched a GitHub integration that scales your Git repos to handle 100 terabytes of files in a single repo.

XetData enables data scientists and machine learning engineers to version code, models, and datasets together. Most teams have glued together clunky workflows using S3, DVC, Git, Git LFS, and other tools and make true reproducibility difficult: https://news.ycombinator.com/item?id=37694701

We instead embrace and extend Git so end-users don’t need to learn a new tool and a new set of commands. Our implementation is similar to Git LFS, where we take over the .gitattributes file, push pointers to large files in GitHub, and push the raw, large files to us. We have a few distinct features that we’re proud of that improve the user experience:

- Our XetData bot comments on your pull requests to provide links to useful dataset views and model diffs. We’re working on rendering these inside GitHub itself using browser extensions.

- Git LFS and similar tools only implement file-level deduplication. We created a new technique called block-based deduplication (published in CIDR’23 conference) specifically for data and ML workflows. The ML lifecycle consists of making lots of iterative changes and our technique helps save storage and time spent downloading and uploading changes.

- You can mount large repos to your local machine using git-xet mount for exploratory work. Individual files that are needed are streamed in just in time behind the scenes. We open sourced our implementation of mount and it was well received here on HN: https://news.ycombinator.com/item?id=37573679

- To give more users access to your data, just add them to your GitHub repo.

This is a beta product and we would love all of your feedback. You can find all instructions to try this out here: https://github.com/apps/xetdata

While we’re in beta, our product is completely free to use. We have a Slack you can join or a GitHub issue tracker.

- Slack: https://communityinviter.com/apps/xetdata/xet

- GitHub: https://github.com/xetdata/xet-tools/issues/

Is it easy to try this out on an existing repo and then uninstall it and go back to using git LFS if I don't like it? Can this be done without messing up my repo?

Good question -- we've tested this a bit and it seems pretty clean.

After removing the app, you can make these small changes we've documented in our uninstall path: https://xethub.com/assets/docs/github-app#uninstalling-the-x...

When you try this out, I'd love to know how much dedupe was possible for your Git LFS files (I'm a co-founder at XetHub).

We also have a writeup on the features and workflow if people want to dive deeper: https://about.xethub.com/blog/xetdata-scale-github-repos-100...

I worked at GitHub and this is exactly what I always wanted as someone working with data and ML artifacts. We even prototyped something with GitLFS once that looked like a very hacky and less well-thought-out version of this. Great work.

Thanks for the kind words! We definitely got inspiration from Git LFS