▲ 31 points
back
11 comments
I've seen so many things disappear off the web that I knee-jerk yt-dlp/wget/DeDRM anything I am interested in or is relevant to my personal culture.
What stinks is I haven't found anything satisfactory in terms of archival software that runs locally. For now, I shove things into folders in external drives, but directory trees are very poor when it comes to categorization and retrieval.
While I haven't personally tried it myself, ArchiveBox seems to be tailored specifically for archiving[1]. Mb you should give it a try?
[1]https://archivebox.io/
I too am looking for a tool that can decently mirror sites. The next best thing is something like Instapaper or Obsidian Clipper which export copies of a site (into markdown, in the latter's case).
Modern hard drives are now above 20TB in capacity. If your average file size is 100KB, then they will hold 200 million+ files.
File systems with their directory or folder hierarchy are poor organizers of files. Searching for files is a huge hassle.
File systems, which were designed over 50 years ago when storage was measured in MB, need to be replaced by object stores that are much better at organizing and finding data.
In what way are object stores better than hierarchical filesystems? With links you can create arbitrary indices over the same data with different (or the same) names.
This is what I do. Some 5 million files (3 TB) with hard and soft links everywhere to make the files show up in places I hope to find them again.
Ripgrep Anything or whatever it is called is pretty good for finding things, at least as long as I can restrict the search to a not too huge subset of my archive.
I also have org-mode documents full of links to interesting files, and sometimes to locations within files.
A bit messy, but I do not want to have dependencies on any special software. My oldest files have been with me since I first started hoarding on 360 kB DOS floppies.
With file systems, searching for files (e.g. find all documents) requires traversing the entire directory hierarchy. With hard drives this comes at a cost of many seeks.
The last time I traversed a folder tree with over 100 million files, it took more than 30 minutes to complete.
A properly designed object store can find everything in less than a minute.
I’m reasonable satisfied with Shiori and its text search functionality
Have you looked at Camlistore?
Now known as Perkeep (previously Camlistore)
Perkeep is very interesting on a fundamental level, as a concept and as a building block, but sadly hasn't received enough love to fully fulfill its vision.
I wish there were more people organizing around Perkeep and breathing life into it.