• WaterWaiver@aussie.zone
    link
    fedilink
    English
    arrow-up
    37
    arrow-down
    1
    ·
    edit-2
    11 months ago

    4.5PB holy shit. You need to stop using UTF2e32 for your text files.

    I’d be paranoid about file integrity. Even a 0.000000000022% (sic) chance of a single bitflip somewhere along the chain, like a gentle muon tickling the server’s drive bus during the read, could affect you. Did you have a way of checking integrity? Or were tiny errors tolerable (eg video files)?

    • quinkin@lemmy.world
      link
      fedilink
      English
      arrow-up
      22
      ·
      11 months ago

      They were using rclone so all of the transfers would be hash checked. Whether the file integrity on either side of the transfer could be relied upon is in some ways a matter of faith, but there a lot of people relying on it.