We built a Rust implementation of NFS to accomplish the same thing!
https://news.ycombinator.com/item?id=37573679
This lets you:
- Mount a Git repo + branch state to a folder: `git xet mount https://xethub.com/XetHub/LAION-400M.git --prefetch 0`
- Analyze parquet files using DuckDb:
`import duckdb`
`duckdb.query("select COUNT() from 'data/.parquet'")`