Backup for important files/pictures?
-
How do you backup important things you store in selfhosted clouds?
I’m currently thinking about hosting Ente myself for syncing all my pictures. Maybe also spinning up nextcloud for various other shared files.
However, for me one main benefit of using services like iCloud is the mitigated risk of losing everything in case the hardware fails (and fire, theft, water-damages, …).Do you keep regular updates on hosted services? Do you keep really important stuff on other providers? Do you have other failsafes?
I use Backblaze personal/unlimited, and have for quite a while. A lot of the other storage options go by GB/price which is fine, but I have a ton of stuff that is irreplaceable such as my music collection of around 80k songs I converted out to flac, pictures, business docs, etc. I realize it's not really in the selfhosted arena, but Backblaze works out for me. If you are backing up a lot of data, re-initializing multiple TB backups can be a chore. Backblaze has a program where you buy a 10 TB drive from them, they ship you your data, once transferred you can send the drive back for a full refund.
-
I use a 48TB ZFS RAIDZ2 pool to maintain data integrity locally and keep rolling ZFS snapshots with sanoid so that data recovery to any point within the last month is possible. Then I use borgmatic (borg) to sync the important data (~1TB) to a Servarica VPS (Polar Bear plan, which works out to be cheaper than Backblaze B2 costs for my purposes). The Servarica server really sucks in terms of CPU, and it's quite sluggish, but it's enough for daily backups. I also self-host healthchecks.io on a free Fly.io VPS thing (not sure if they offer this anymore) to make sure the backups are actually happening successfully, and hosting that on a third-party VPS means that it's not going to fail at the same time my server does. Then I use Uptime Kuma to make sure everything is consistently alive (especially the healthchecks.io server, which in turn verifies that Uptime Kuma stays alive). I also run the same borg configuration to back up to a plain non-redundant disk locally.
The downside of this setup is that I'm only truly backing up a fraction of my pool, but most of my pool is stuff that I can redownload and set up again in the event of e.g. a house fire. I also run a daily script to dump a lot of metadata about my systems and pool, like directory listings of my media folders and installed programs/etc, which means that even if the data might be lost, I have a map of what I need to grab again.
How big are these ZFS snapshots compared to the stored data size? 1:1?
-
How big are these ZFS snapshots compared to the stored data size? 1:1?
Snapshots basically put a pin in all the data at that moment and say "these blocks are not going to be physically deleted as long as I exist", so the "additional" data use of the snapshots is equal to the data contained within the snapshot that doesn't exist at the current moment. I.e., if I have two 50GB files, take a snapshot, and delete one, I will still have 100GB physical disk usage. I can also take 400 more snapshots and disk usage will remain at 100GB, as the snapshots are just virtual. Then I can either bring that deleted file back from the snapshot, or I can delete the snapshot itself and my disk usage will adjust to the "true" 50GB as the snapshot releases its hold on the blocks.
What sanoid and other snapshot managers do is they repeatedly take snapshots at specified intervals and delete old snapshots past a certain date, which in practice results in a "rolling" system where you can access snapshots from e.g. every hour in the past month. Then once a snapshot becomes older than a month, sanoid will auto-delete it and free up the disk space that it's holding onto. The exact settings are all configurable to what you're comfortable with in terms of trading additional physical disk usage of potential "dead" data for the convenience of being able to resurrect that data for a certain amount of time.
I really like the "data comet" visual from this Ars Technica article.
-
Obligatory: RAID is not a backup.
I'll make sure to follow the 3-2-1 principle as I move off of the cloud
-
How do you backup important things you store in selfhosted clouds?
I’m currently thinking about hosting Ente myself for syncing all my pictures. Maybe also spinning up nextcloud for various other shared files.
However, for me one main benefit of using services like iCloud is the mitigated risk of losing everything in case the hardware fails (and fire, theft, water-damages, …).Do you keep regular updates on hosted services? Do you keep really important stuff on other providers? Do you have other failsafes?
wrote on last edited by [email protected]500 gigs in hdds i ripped of cable boxes all wired together an hooked up to my old thinkpad t470 running thunar
-
How do you backup important things you store in selfhosted clouds?
I’m currently thinking about hosting Ente myself for syncing all my pictures. Maybe also spinning up nextcloud for various other shared files.
However, for me one main benefit of using services like iCloud is the mitigated risk of losing everything in case the hardware fails (and fire, theft, water-damages, …).Do you keep regular updates on hosted services? Do you keep really important stuff on other providers? Do you have other failsafes?
wrote on last edited by [email protected]PC: Veeam
Phone/general pics: Immich (both automatic and manual)
Some general phone files: Syncthing
The remaining stuff is on my NAS at home.
Off-site of the most important VMs and some infrastructure: Veeam Backup Copy to an external HDD I keep at my workplace (encrypted) -
I have 5 copies of all my files on 5 devices, synced using syncthing with staggered file versioning. 2 of those are with friends and family who let me put a thin client at their place.
To protect against me misconfiguring syncthing, or some bug deleting all copies, every 3 months I manually make a copy and put it on a hard drive into a fire resistant safe.
While impressive this seems like such a hassle to keep up
-
While impressive this seems like such a hassle to keep up
The manual copy is a bit annoying, but in the end it's maybe 10 minutes of work. Start the transfer in the evening, it's finished in the morning.
-
I use nextcloud and I love it.
You want to follow the 3-2-1 strategy: 3 copies of your data on at least 2 different forms of media, and 1 backup being off-line.
Off-site rather than offline, protecting against things like your house burning down.
-
How do you backup important things you store in selfhosted clouds?
I’m currently thinking about hosting Ente myself for syncing all my pictures. Maybe also spinning up nextcloud for various other shared files.
However, for me one main benefit of using services like iCloud is the mitigated risk of losing everything in case the hardware fails (and fire, theft, water-damages, …).Do you keep regular updates on hosted services? Do you keep really important stuff on other providers? Do you have other failsafes?
I use:
- 3-way zfs mirror for important data like photos, documents
- snapraid for bulky and less important data like movies
- hourly backup of important data and a subset of the less important data (difficult to find movies) to a rpi with a big disk
- daily backup of the same data to a friend. We have a system where we put a hdd in each other's server and have ssh access
Backups are done using restic
-
How do you backup important things you store in selfhosted clouds?
I’m currently thinking about hosting Ente myself for syncing all my pictures. Maybe also spinning up nextcloud for various other shared files.
However, for me one main benefit of using services like iCloud is the mitigated risk of losing everything in case the hardware fails (and fire, theft, water-damages, …).Do you keep regular updates on hosted services? Do you keep really important stuff on other providers? Do you have other failsafes?
When I setup my backup strategy, the Hetzner storage boxes showed the best price per TB, and allow ssh / SCP and webdav access.
I run a virtualized TrueNAS Scale with a passed through HBA and about 6 drives in raid-z2 mode. I setup TrueNAS to do regular snapshots - hourly, daily, monthly and I use the built in encrypted backup options to backup my most important data every night (personal videos and pictures, VM backups, documents, docker volumes).
This works like a charm and it seems to be also quite stable. If the system ever dies, I can always simply mount the encrypted backup folder on my laptop using rclone, and then manage my files directly using mc.
