I have a server running Debian with 24 TB of storage. I would ideally like to back up all of it, though much of it is torrents, so only the ones with low seeders really need backed up. I know about the 321 rule but it sounds like it would be expensive. What do you do for backups? Also if anyone uses tape drives for backups I am kinda curious about that potentially for offsite backups in a safe deposit box or something.

TLDR: title.

Edit: You have mentioned borg and rsync, and while borg looks good, I want to go with rsync as it seems to be more actively maintained. I would like to also have my backups encrypted, but rsync doesn’t seem to have that built in. Does anyone know what to do for encrypted backups?

  • Deckweiss@lemmy.worldEnglish
    11·
    11 months ago

    The software borgbackup does some insane compression.

    It is more effective if you backup multiple machines tbh (my 3 linux computers with ~600gb used each get compressed down to a single ~350gb backup, because most of the files are the same programs and data over and over again)

    But it might do a decent enough job in your case.

    So one of the solutions might be getting a NAS and setting up borgbackup.

    You could also get a second one and put it in your parents or best friends home for an offsite backup.

    That way you don’t have to buy as large of a drive capacity, but will only have fixed costst (+electricity) instead of ongoing costs for some rented server storage.

    I guess that would be about 400$ per such a device, if you get a used office pc and buy new drives for it.

    Tape seems to be about half the price per TB, but then you need special reader/writer for it, which are usually connected via SAS and are FUCKING EXPENSIVE (over 4000$ as far as I can see).

    It only outscales HDDs in price after like ~600TB

  • solrize@lemmy.worldEnglish
    7·
    11 months ago

    I’ve been using Borg and Hetzner Storage Box. There are some small VPS hosts that actually beat Hetzner’s pricing but I have been happy with Hetzner so am staying there for now. With 24TB of data you could also look at Hetzner’s SX64 dedicated server. It has a 6 core Ryzen cpu and 4x 16TB HDD’s for 81 euro/month. You could set it up as RAID 10 which would give you around 29 TiB of usable storage, and then you also have a fairly beefy processor that you can use for transcoding and stuff like that. You don’t want to seed from it since Hetzner is sticky about complaints that they might get.

    Tape drives are too expensive unless you have 100s of TB of data, I think. Hard drives are too unreliable. If you leave one in a closet for a few years, there’s a good chance it won’t spin back up.

  • ramble81@lemm.eeEnglish
    6·
    11 months ago

    I have my BD/DVD/CD collection backed up to S3 Glacier. It’s incredibly cheap, offsite, and they worry about the infrastructure. The amount of Hard drive and infrastructure space you’ll need to back up nearly that amount will cost you the about the same give or take. Yes it’ll cost a bit in the event of a catastrophic restore, but if I have something happen at the house, at least I have an offsite backup.

  • taladar@sh.itjust.worksEnglish
    5·
    11 months ago

    I have just been using Borg with a Hetzner Storagebox as the target. That has the advantage of being off-site and not using up a lot of space since it deduplicates. It also encrypts the backup. It might take a while for the initial backup at 24TB though depending on your connection.

    • poncho@lemmynsfw.comEnglish
      1·
      11 months ago

      Damn never heard of them looks great. Is there any catch or is it like a small company that might go out of business in a few years? I still haven’t had to backup more then 4tb but once I do get up to those numbers they might be the best option compared to offsite hard drives like I been doing

      • taladar@sh.itjust.worksEnglish
        2·
        11 months ago

        They are anything but small. They are probably one of the biggest German hosting companies out there.

      • qaz@lemmy.worldEnglish
        1·
        11 months ago

        I have been using their nextcloud service for several years now and it works great.

  • ancoraunamoka@lemmy.dbzer0.comEnglish
    61·
    11 months ago

    I am simple man s I use rsync.

    Setup a mergerfs drive pool of about 60 TiB and rsync weekly.

    Rsync seems daunting at first but then you realize how powerful and most importantly reliable it is.

    It’s important that you try to restore your backups from time to time.

    One of the main reasons why I avoid softwares such as Kopia or Borg or Restic or whatever is in fashion:

    • they go unmantained
    • they are not simple: so many of my frienda struggled restoring backups because you are not dealing with files anymore, but encrypted or compressed blobs
    • rsync has an easy mental model and has extremely good defaults
    • mea_rah@lemmy.worldEnglish
      3·
      11 months ago

      FWIW restic repository format already has two independent implementations. Restic (in Go) and Rustic (Rust), so the chances of both going unmaintained is hopefully pretty low.

    • HumanPerson@sh.itjust.worksOPEnglish
      1·
      11 months ago

      I was heavily considering borg but I just looked up rsync and it looks like everything I need. Thank you.

      Edit: Actually encryption would also be nice. Is there any way to do that with rsync?

      • ancoraunamoka@lemmy.dbzer0.comEnglish
        1·
        11 months ago

        what other people are saying, is that you rsync over an encrypted file system or other type of storages. What are your backup targets? in my case I own the disks so I use LUKS partition -> ext4 -> mergerfs to end up with a single volume I can mount on a folder

    • ShortN0te@lemmy.mlEnglish
      1·
      11 months ago

      One of the main reasons why I avoid softwares such as Kopia or Borg or Restic or whatever is in fashion:

      • they go unmantained
      • they are not simple: so many of my frienda struggled restoring backups because you are not dealing with files anymore, but encrypted or compressed blobs
      • rsync has an easy mental model and has extremely good defaults

      Going unmaintained is a non issue, since you can still restore from your backup. It is not like a subscription or proprietary software which is no longer usable when you stop to pay for it or the company owning goes down.

      The design of restic is quite simple and easy to understand. The original dev gave multiple talks about it, quite interesting.

      Imho the additional features of dedup, encryption and versioning outweigh the points you mentioned by far.

      • ancoraunamoka@lemmy.dbzer0.comEnglish
        11·
        11 months ago

        Going unmaintained is a non issue, since you can still restore from your backup. It is not like a subscription or proprietary software which is no longer usable when you stop to pay for it or the company owning goes down.

        Until they hit a hard bug or don’t support newer transport formats or scenarios. Also the community dries up eventually

        • ShortN0te@lemmy.mlEnglish
          0·
          11 months ago

          Until they hit a hard bug or don’t support newer transport formats or scenarios. Also the community dries up eventually

          That is why you test your backuo. It is unrealiatic, that in a stable software release there is suddenly, after you tested your backup a hard bug which prevents recovery.

          Yes unmaintained software will not support new featueres.

          I think you misunderstood me. You should not use unmaintained software as your backup tool, but IMO it is no problem when it suddenly goes unmaintained, your backup will most likely still work. Same with any other software, that goes unmaintained, look for an alternative.

          • ancoraunamoka@lemmy.dbzer0.comEnglish
            1·
            11 months ago

            It is unrealiatic, that in a stable software release there is suddenly, after you tested your backup a hard bug which prevents recovery.

            How is unrealistic? Think of this:

            • day 1: you backup your files, test the backup and everything is fine
            • day 2: you store a new file that triggers a bug in the compression/encryption algorithm of whatever software you use, now backups are corrupted at least for this file Unless you test every backup you do, and consequently can’t backup fast enough, I don’t see how you can predict that future files and situations won’t trigger bugs in a software
            • ShortN0te@lemmy.mlEnglish
              1·
              11 months ago

              We talk about software that is considered stable. That has verification checks for the backup. Used by thousands of ppl. It is unrealistic.

  • hperrin@lemmy.worldEnglish
    4·
    11 months ago

    I have a machine at my parents’ house that has a single 20TB drive in it. I’ll log in once in a while and initiate an rsync to bring that up to current with my RAID at home. The specific reason I do it manually is in case there’s a ransomware attack. I won’t copy bad data. That’s also the reason I start it from the backup machine. The main machine doesn’t connect, the backup machine does, so ransomware wouldn’t cross that virtual boundary.

  • narc0tic_bird@lemm.eeEnglish
    4·
    11 months ago

    I backup my /home folder on my PC to my NAS using restic (used to use borg, but restic is more flexible). I backup somewhat important data to an external SSD on a weekly basis and very important data to cloud storage on a nightly basis. I don’t backup my *arr media at all (unless you count the automated snapshots on my NAS), as it’s not really important to me and can simply be redownloaded in most cases.

    So I don’t and wouldn’t apply the 321 rule to all data as it’s simply too expensive for the amount of data I have and it’d take months to upload with my non-fiber internet connection. But you should definitely apply it to data that’s important to you.

  • capital@lemmy.worldEnglish
    3·
    11 months ago

    My use case is basically the same as yours.

    I do restic to Wasabi.

    I’ve been on restic for a few years now and have never had an issue. I started out using Google Drive for the backend but that was though my college which went away eventually so I swapped over to Wasabi but I’m considering B2.

    It’s actively maintained and encrypted.

    There are a handful of backends it supports but can be extended by writing to an rclone backend.

    • tal@lemmy.todayEnglish
      0·
      11 months ago

      Synology Nas(12TB raid 1)

      I have to say that I was really surprised that apparently there isn’t a general solution for gluing together different-sized drives in an array reasonably-efficiently other than Synology’s Hybrid RAID. I mean, you can build something that works similarly on a Linux machine, but there apparently isn’t an out-of-the-box software package that does that. It seems like the kind of thing that’d be useful, but…shrugs

  • rambos@lemm.eeEnglish
    31·
    11 months ago

    I use Kopia to backup all personal data (nextcloud, immich, configs, etc) daily to another disk in the same server and also to backblaze B2. Its not proper 321 but feels good enough. I dont backup downloadable content because its expensive

  • sloppy_diffuser@sh.itjust.worksEnglish
    2·
    11 months ago

    Important stuff (about 150G) is synced to all my machines and a b2 Backblaze bucket.

    I have a rented seed box for those low seeder torrents.

    The stuff I can download again is only on a mirrored lvm pool with an lvmcache. I don’t have any redundancy for my monerod data which is on an nvme.

    I’m moving towards an immutable OS with 30 days of snapshots. While not the main reason, it does push one to practicing better sync habits.

  • douglasg14b@lemmy.worldEnglish
    2·
    11 months ago

    I might be crazy but I have a 20TB WD Red Pro in a padded, water proof, locking, case that I take a full backup on and then drive it over to a family members 30m away once a month or so.

    It’s a full encrypted backup of all my important stuff in a relatively different geographic location.

    All of my VM data backs up hourly to my NAS as well. Which then gets backed up onto the large drive monthly.

    Monthly granularity isn’t that good to be fair but it’s better than nothing. I should probably back up the more important rapidly changing stuff online daily.

  • kylian0087@lemmy.worldEnglish
    1·
    11 months ago

    What I use is Borg. I use Borg to backup the server to a local NAS. Then I have a NAS at my grand parents house which I use to store the backups of the NAS it self.

  • douglasg14b@lemmy.worldEnglish
    1·
    11 months ago

    As of today I’m actually in a lucky position where I am now able to set up a secondary NAS at my brother in laws and use that as a backup server that I can back up to essentially in real time.

    All it’ll cost me is the hardware and the electricity.