We have several layers of backups for our data. BigData (drive enclosure T) is a 2.7 TB drive with backups for a week. tempest (drive enclosure V) has backups for 90 days (it’s useful to have incrementals). There’s also a 3 TB drive (drive enclosure U) with a copy of the whole machine (in case of hackers, fumbles, or hardware failures). Last week, Erin brought up that she’d been receiving several emails about “/root/snapshots not mounted!” The cause of this was a backup scare: both tempest and the 3TB drive could not be read by their drive enclosures. It was thought for a few days that we’d lost all those backups…
Forumgoers suggested that the problem might be with the quality of the drives (consumer drives are 3x cheaper than more quality drives, so we use the cheap ones), or that moving the drives in and out of the cabinets wasn’t a good idea… or that something could be wrong with their drive enclosures, which turned out to be correct. Luckily, when they were read by T instead, it turned out that the drives weren’t unreadable and everything was fine.
Unfortunately, we’re still getting kernel errors and the like sometimes from the drives, but at least we have backups for now.