▲ 115 points
back
82 comments
Two additional suggestions:
1. Test your backups. I see no testing procedure described, and although maybe the author's scripts have spot checking beyond what is described in the article, code will only ever catch the cases it was programmed to expect.
Manual testing does not have to be frequent or extensive, but making sure the backups are still recoverable and complete (take five minutes to spot check some old and new files) every six months helps ensure no silent issues are occurring
2. Less importantly: use append-only backups to an independent system. While ransomware for consumers is comparatively rare these days, it would give me peace of mind. The author's deposit box mostly fulfils that purpose, but whether it gets filled with bogus data or wiped by the malware is a matter of luck in timing (with the odds heavily in their favor due to the irregular connection of the drive and the existence of another computer that the malware probably isn't programmed to expect).
Of course, "append only" is a bit of a misnomer because eventually you'd run out of space, but the system should independently delete old data rather than being able to command it remotely (from the potentially infected system) to wipe anything. Restic has docs with considerations for data deletion in an append-only setup (disclose: I helped write them), which are also relevant if you don't use restic because they describe the method more than the specific implementation: https://restic.readthedocs.io/en/latest/060_forget.html#secu...
Doesn’t sound like backing up, but keeping your stuff massively in sync.
What happens if you discover something is missing or corrupted after 6 months?
If it's not something super critical in my experience the simpler solution is better for longterm or fool-proof backups.
E.g. if I use some special backup software that creates incremental archives or whatever, and then I die... My family would have absolutely no idea how to access that data. While with a normal NTFS partition and data on an external HDD they could just plug it in some PC and look at the family photos.
The ZFS clone should deal with that via regular snapshots.
> here is how I do my backups. It takes me about ten seconds per day and five minutes per month to maintain
> rsync everything to each of them, bringing one down to a safe deposit box downtown, and taking out the one that was there from my last visit
Sounds like you are underestimating how much time you spend on backups. Sounds good though
I assumed the author has to make the trip to the bank anyway. I integrate my off-site, offline backup storage with visits to my parents, for example.
My absolute final resort "solution" for data loss of keepsakes:
Accept the impermanence of data, memories and memorabilia.
Ideally I'll strive to keep that data backed up as long as I can, and will mourn its loss, but mentally I prepare myself that one day I might lose access in some way. Most importantly, that data is also 99% irrelevant to the next generation, maybe save for a handful of photos or videos ("what did grandpa look like?"). So in the grand scheme of things it's not like I have to be an epic archivist of my own life.
I suppose it helps that this was done for me in the past - toxic family members threw out all my childhood family photos a decade ago.
"Toxic" seems quite the understatement! How sad.
I went to school with a boy who had no photos from his youth. His parents took many photos but didn't get them developed, and when they took all the films to the pharmacy (here in the UK) to get them developed, the pharmacy sadly burned down and they didn't ever get any photos. Quite sad I thought.
I use a similar, but I think simpler, setup.
"Files I use and change every day" are on Dropbox, which gets synced to a NAS.
Photo and video files are directly stored on the NAS and don't live anywhere else.
The NAS is rsynced automatically every day to an external disk (connected via USB) of which there are three copies. One is in a safe at home, the other at my parents' house. The one in the safe is rotated every week, the one at my parents' every month.
If my house gets robbed or even burns down it's likely the disk in the safe will survive and I lose only one week of data. If the house gets completely destroyed or the safe is cracked, then I can turn to the disk at my parents and lose one month.
It's true that rsync is not a backup. But by default rsync only adds data. Delete is an option one has to turn on. My daily rsync does not have the delete option. I have a different script with delete, that I run manually.
The theory was that I would review the delete process with a dry run before committing to it. In practice I don't do this and simply run the delete script occasionally, which ruins the purpose. But I guess I could be more careful if I really wanted to.
I've been happy with: https://kopia.io/
Fairly easy to configure, does snapshots to S3 and has a icon in my tray I can watch :)
It seems that Kopia doesn't support continuous backup, which may be a requirement for some users (detecting changes on a large number of files is resource-intensive).
kopia is one of those things are forever in my todo list while i use rsync daily.
Interesting. Just for sharing, for backup on Linux I've been using rsnapshot, restic (which accepts Rclone as interface) and ReaR (Relax and Recover) for full linux recovery; on Windows, commercial Acronis and recently UrBackup. Old good "pathsync" from Cockos also helps on LAN.
Everything boils down to your amount of files or GBs when some solution becomes "too slow" doing incrementals and/or file processing. My two cents.
On Linux: https://www.borgbackup.org/
> Whenever I turn on one computer, I sync it from the other.
> Daily rsync to an encrypted Linux USB stick attached to the desktop.
> Daily rsync to an encrypted MacOS USB stick attached to the laptop.
> Daily rsync to an encrypted ZFS SSD inside the desktop.
> Daily rsync to OpenBSD remote attached storage at vultr.com
> [...] I do it a few times a day, especially if I’ve just made or saved something of value. And always right before I shut down the computer, which I do almost any time I step away from it for more than a few minutes.
Have you ever noticed blog posts rarely go viral if they describe a normal setup, and are much more likely to do so if they describe a really complicated setup?
I'm all for people coming up with needlessly complex homelab setups if they have fun doing so - but sometimes I wonder whether newbies see this sort of thing, and think Linux needs this level of complexity.
> I wonder whether newbies see this sort of thing, and think Linux needs this level of complexity.
To be fair, people who aren't already familiar with Linux, won't even come across these articles to begin with.
But I can see when this type of articles reach someone through a network on Twitter or reddit, when they hear alien technical tools and script this and that they think: "Cool, that's what IT specialists do, not for me, I'm doing fine."
What complexity? rsync to 4 locations, likely from some script?
> Drive downtown to a safety deposit box.
Resilient? Yes. Simple? No.
Question for rsync users on linux. Does your rsync preserve creation times --crtimes ?
I'm aware ext4/3 filesystems don't store creation times. But I want to use rsync on Linux to sync files between an NTFS(supports creation times) and a Btrfs partition(also supports creation times) without losing the creation times.
Currently rsync just sets the creation time to the modification time on the destination directory. Which is not what I want.
When I use `--crtimes` I get a "This rsync does not support --crtimes (-N)" error.
Weirdly, just using `cp -a` works. But it doesn' t have any of the checksumming and differential copy bells and whistles of rsync.
How do you rsync on linux while preserving creation times(on supported file systems)?
I just migrated from a QNAP NAS to a custom build with ZFS. I have files with creation times dating back to the 90s on my original FAT16 filesystem and I didn't want to lose them, so I recently did a (way-too-)deep dive on this, I'll summarize my notes for you.
First, ext4 actually does support creation times, called "crtime". But there's some internet confusion about it since this support predated linux kernel support, so you had to use ext4-specific tooling (on an unmounted filesystem) to access it, e.g.:
debugfs -R 'stat <inode>' /dev/nvme0n1p1
debugfs -R -w 'set_inode_field <inode> crtime <new-time>' /dev/nvme0n1p1
But the normal file syscalls only support reading btimes, not setting them to arbitrary values. And rsync on linux, as you saw, can't do anything the kernel doesn't have a syscall for. For a while the only option was to:
1. set the system clock
2. create a file (at which point the kernel sets btime to the system time, plus a few nanoseconds)
3. restore the system clock
Obviously a huge hack, and needs root, but tools like s-tar automated it (search for "time storm" in this manpage):
https://web.archive.org/web/20220331080358/http://schilytool...
I almost gave up and spun up a Windows VM (since Windows has supported reading and writing creation times since the beginning). But then it clicked -- the kernel interface to the filesystem module takes the btime as an explicit parameter. So if you could find a kernel module that talks to the filesystem module directly (instead of going through the usual high-level file syscalls), you can pass along any btime you want. And there just so
I use Syncthing on my PCs and NAS, and Backblaze to trivially do a 3-2-1 backup strategy.
My work directories are synced on all devices. With my NAS up 24/7, I can always pick up the last changes when I switch computers. The irreplaceable documents (all my notes, taxes, company accounting,...) are also backed up by the NAS on Backblaze, keeping all versions. I occasionally create a snapshot on Backblaze for those files – should totally automate that but I'm lazy.
I do agree though: it's more "syncing stuff between multiple places" than a real backup.
I keep my important files on my DIY NAS -- a cheap NUC with two external HDDs (Btrfs in RAID1). Every night at 1am a cron job takes a restic snapshot to a Backblaze bucket. I feel like bang for buck my solution isn't half bad. The NUC is also my home server (web, Pi-hole, Home Assistant, ...), so it was already there, the HDDs were cheap and I currently pay less than $3 a month for over 300 GB of data in the bucket.
I do something similar[1] with rsync --link-dest and hard links, so that I can go back in time if I delete something. OP seems to just have a copy of his actual files, but if he sees a file missing after 2 weeks after deleting it, he has no way of recovering it.
Or if you don't want to write a DIY solution you could use this (shameless plug): https://rsync-backup-ras.sourceforge.io/
Everything that matters is checked into fossil. Box hosting that uses zfs. That rules out most failure modes other than gross incompetence, partially mitigated by a periodic backup to external drives.
In the before times I put everything in Dropbox. Dropbox synced with an empty directory by mistake and deleted everything. Not going to use Dropbox again.
I'm using :
- Dropbox for "core files" (small subset of files I don't want to lose, mostly my notes, papers, etc) - Arq with S3 for backups (do a backup each hour) - An external SDD to do semi-frequent backups (mostly used in case I change laptop to ensure I have a backup of my files, but otherwise I mostly rely on Arq)
Blank lines prevent the bullet points running together
I use Vorta. I have two profiles, one for my home dir, and recently I added one for all dotfiles I would want to keep when setting up a new machine. It's only about 5 files but it avoids using yet another tool for dotfiles.
Are backup stored to Glacier is my house-burned-down archive for about $5/mo. It may cost a lot to get out, but at that point our family pictures and movies are gone anyway, so it would be worth the cost.
Thanks for sharing! I'm going to go and test my backups now...
that's a somewhat expensive setup to still do all the work of dealing with dead drives and manually doing replication and dealing with conflicts (I fear the pain of dealing with his 4way daily backup of the same data over desktop/laptop/2 pen drives)
zfs rent = 4+TB drive + $10/mo (base 1TB) + $5*3 (extra 3TB) = drive + $25/mo
hetzner = 5TB $10.9/mo
vultr = $5/mo (probably more)
4x ~4TB drives = $800
assuming you replace drives every 5yrs: $3,254 or $54.23/mo.
assuming the drives last 10yr which are common now: $5,708 or $47.56/mo.
A 4tb drive costs 80-120€
7€/month for 5y
Anyone know/recommend something like ZFS.rent in Europe?
I use rsync.net, but I don't know if their `zfs receive` is encrypted:
rsync for backup is a recipe for disaster if you don't realize you just rsynced and overwriten good data with corrupted or ransomwared data.
If you want to use rsync, do rsync and snapshots...and those snapshots should automatically made on the remote destination (can be just a daily cron), only accessible read only from the source machine without having any administrative rights to tamper them.
Author: describes a simple setup which probably puts him in the top 0.1% of personal users in terms of resilience to data loss.
HN: nitpick about potential problems which are present even in some industry grade data loss solutions.
Please make your substantive points without snark or swipes.
This is in the site guidelines: https://news.ycombinator.com/newsguidelines.html.
"data loss solutions" is a term/product category I can totally get behind :D And indeed, there's planty of those in the IT industry!
And honestly, I never lost data in my whole life as a private user. Exceptions: Own stupidity in which case even back-ups don't help.
And that included the time when HDDs had a tendency to fail. And what little data loss occured was actually a grand total of 3 hours of work to recreate it.
I simply don't get the obbsession people have with data back ups for their private needs.
Here I was starting to think 2 USB (platter) drives and the occasional rsync -uavPi is too much.
All power to the author, that's a hell of a set up.
My setup :
- servers and laptops (laptops : except /home): restic to local minio instance + rclone to B2 storage
- /home of laptops : kopia to b2 storage
I'm using rdiff-backup to have versioned copies, however at some point of time trimming of the snapshots is required
I do a zfs snapshot and than do a sync.
I never delete those snapshots because the big items like images don't change.
I've read USB sticks and backup in the same phrase, something must be wrong.
Given that it's just one out of several backup media, that's kinda fine, I guess? Just don't put all your eggs in one basket. Which is why I'd add at least one HDD to the mix, as all the local options appear to be flash-based. Flash memory can develop defects especially if it's not turned on for a while. For long-term backup storage, a HDD might be a better solution.
I've used Backblaze for years and it's worth the money. I've only done a full restore from it once after a laptop was bricked and it was seamless. There have been a few times that I needed a recently deleted file and it worked for that too.
Ah yes, HN's automatic title editorializing strikes again, removing 33% of the title's wordcount and changing it from "click here to see my backup method" to "hey, I do backups".
> I Backup
Well, that's a fun casualty of the title filter!