I was speaking with a colleague last week about UK businesses scaling back their budgets for disaster recovery and business continuity provision. It seems that while some firms have decided to take the risk of not having a plan at all, others are trying to find shortcuts to reduce their spending. By far the most obvious piece of the jigsaw to remove, for most businesses, are the test invocations.
Test invocations form a crucial part of all disaster recovery plans, but often it is the most expensive component of the solution. Test invocations are frequently overlooked at the outset of a business continuity plan, as service providers and manufacturers proclaim ‘ease of recovery’. Only when the first test is carried out does the extent of the hidden costs become apparent. Even simple tape restore testing can be time consuming and therefore expensive (and often outside the desired Recovery Time Objective or RTO). Worse still, if the test fails, further staff time must be dedicated to investigation and documentation updates. When job losses are on the horizon, and teams are running on empty, just sparing the staff to fulfil the project may not be an option.
Some DR processes have an even higher cost due to bad design, and can only be carried out at the expense of uptime. Physical servers sometimes need to be moved, or shutdown to carry out all the environment or application testing. Some business continuity advisors get it right and ask service providers to ‘bundle’ test invocations into the service contract. That is fine as far as it goes, but it still frequently does not account for the hidden costs like resource, transport, and documentation updates.
It seems fair then to reduce or postpone test invocations as part of a budget cutting directive, but at what cost? When times are good, and business is booming, cash flow is rarely a problem. IT budgets increase as stakeholders recognise the need for business continuity plans and related insurance strategies. In reality, during such times, the organisation may be able to recover from the impact of a couple of days IT downtime. Sure, some customers will switch to your competitors, some of those will never come back, but your order book, and cash flow will be strong enough to carry the business through. In contrast, during a recession, when order books are small, and cash flow is tight, the same period of IT downtime, and resultant loss of business, could be enough to break the camel’s back. Hence, economic recession makes a working business continuity plan even more crucial.
Some service providers, like virtualDCS (but there are others), have engaged with their customers to find a solution to this dichotomy. It is possible, given the right approach, to leave the invocation process to the service provider. The service provider maintains a detailed documentation process, and provides both the equipment and the manpower to invoke the solution independently, with no impact on the client’s live running IT operation, or the team supporting it. Once the solution is fully invoked, the business can carry out specific application tests, before leaving the service provider to dismantle the invocation test again, and update the documentation.
This sounds like a shift to wholly outsourcing the disaster recovery solution to a service provider, and it is. It also sounds very expensive, but it isn’t. The recovery team at virtualDCS (I can’t speak for other service providers), perform test invocations every day of the year. Fortunately live invocations are rare, but test invocations happen on a regular basis. Because the test invocation is a routine action, and highly automated, the costs are kept small, and more importantly, included in the contract. With contracts starting at around £50/week for a server with 60GB of data, and an achievable Recovery Point Objective (RPO) of near zero, why would you do it yourself?