Since writing and speaking a bit more about the “relax! a failure is not an emergency” concept, more and more people approach me with interesting horror stories. I’m scribbling a few backup-related ones here for your enjoyment – and naturally there are important lessons.
Story 1: A place makes backups that get shipped off-site, interstate even. One day a couple of files are lost, and so someone files a request to retrieve said files from the archive. Well, apparently that’s something that should be done as it creates some very stressed responses and a quoted timeline of a few weeks. In the end the issue is resolved through other means and the request stopped – unfortunate, since it would have been very interesting if the requested files would actually ever arrive… clearly a retrieval was not part of the expected process. One also wonders how long a full dataset retrieval would take, or if it’s even possible!
Story 2: A place has data recovery/failover infrastructure. However, it’s in use as a test environment and consequently an actual event that requires use of this hardware would first need re-imaging of the boxes to get them set up to even receive the data they need to contain. Estimated timeline: days to weeks.
The aboves are extreme, but I think they make the point pretty well. I hear and see lots of cases where there are processes in place for making backups, and off-site transfer. Lovely. However, they do often appear to “forget” the objective of which these processes are merely a part, which is not just shippping out a bucket of bits but potentially using it for failover or recovery!
So you want to test the recovery process, that is the entire trail from the storage all the way to having an actual new functioning copy. And, you want to see how much time that takes.
In Open Query parleance, if you haven’t tested this fully, or the time required is more than the downtime your business can afford, then we say you technically have no backup. Simple. No sense being fuzzy about this, right?
It’s one of the things we help our clients with as part of our service. This is not really a topic that should be regarded as optional.