by Scott Owens

In the past five years, media coverage of disasters, both natural and man-made, has shown us the absolute devastation that is possible. Hurricanes, tornadoes, tsunamis, earthquakes, and terrorism command the headlines in newspapers and on TV. Lives are lost, homes and businesses are destroyed, and the ability of relief organizations to provide service is impaired.

The effect is multiplied when the disaster strikes a health care organization that is expected to be part of the first response or to care for the sick and injured. While most health care organizations will not experience the direct effect of such terrible events, lesser disasters happen every day. Even a “minor” disaster can have a large impact, because any interruption of service delivery is costly from both a human and a monetary perspective.

How costly? Consider these eye-opening statistics:

• In an average 500-bed hospital, every hour of downtime costs more than $15,840, according to a Healthcare Informatics article from April 2002 (citing a 2001 survey by the Healthcare Information and Management Systems Society).
• Two out of five businesses that experience a major disaster will go out of business within five years, according to a Gartner Group article from 2001.
• It takes an average business one full month to recover from a computer virus disaster at an average cost of $130,000, according to the 2004 Computer Virus Prevalence Survey by ISCA Labs.
• In April of 1998, a 26-hour frame relay network outage at AT&T resulted in tens of millions of dollars in rebates paid to customers with service-level agreements.
• Within one month of hurricane Katrina’s 2005 landfall in New Orleans, the local electric utility, Entergy Corp., filed for bankruptcy, according to a Wikipedia report.

Even so, maybe you’re still convinced that a disaster won’t happen to your facility, or if it did, that it wouldn’t have a huge impact. Or perhaps you simply haven’t thought about it.

Make 2006 the year of the plan

Why plan? For the same reason athletes practice, soldiers train for battle, and first-time expectant fathers carefully map out the route to the hospital: Because preparation produces better outcomes.

Within health care organizations, Information Services is no longer an 8-5 operation. Clinicians need 24/7 uptime to utilize clinical applications that support computer physician order entry (CPOE), access to patient care history, and EMR documentation of patient care delivery. There is an ever-increasing need for redundant networks, application servers, and data to minimize downtime and ensure performance. Clinicians anticipate uninterrupted access to patient information and clinical decision support tools in providing safe and effective care to patients.  

How do you know if your business is ready to handle a disaster? Contemplate the following questions:

• Is your organization responsible for health or safety of the public at large?
• Do you have any systems that, if they went down, would result in danger to the life or health of employees, patients, or local residents?
• What effect would an emergency have on your ability to conduct business?
• What is your cost of system downtime (productivity loss, revenue loss, damage to reputation, long-term financial impact, loss of customers)?
• Do you have any regulatory or legal requirements for certain levels of operation?
• Are you prepared to deal with the media, your stakeholders, and your employees in the event of a business interruption?
• Do you know which resources are required for you to be operational within 1-3 days of a disaster? Are these resources identified by function? Is there a resource requirement timeline for all critical resources?
• Do you have all of your processes documented? If so, do you have manual processes defined in the event that the automated processes fail?
• Do you have service-level agreements with key vendors to assist in a crisis? Have these agreements been tested?
• Do you have a comprehensive plan to guide the disaster recovery and business resumption process? If so, was the plan tested? Is the plan reviewed periodically to ensure that it is current? Is all of the necessary supporting documentation attached to the plan?
• Does your plan contain a process to declare a disaster, emergency, or business interruption event? Is the notification and escalation list current? Does your plan include emergency evacuation procedures? Is the plan maintained off-site?
• Do you have the entire recovery team identified, including the team leader? Does each member of the team understand their responsibilities and priorities?

If the answers to these questions make you uneasy, 2006 may be time for you to begin planning.

Step 1: Get management on board.

The first step in the process of disaster recovery planning is, without question, gaining the support of executive management. Typically, disaster recovery planning requires the involvement of many people with unique skills from diverse functional areas. It may also require collaboration with other businesses and government agencies.

As the plan evolves, it may also make sense to mitigate some of the identified risks through investments in technology or service agreements with business partners and vendors. The bottom line is that a disaster recovery planning effort must become a formal prioritized project within the organization.

Step 2: Perform a business impact analysis.

The next step in the process of beginning to build a disaster recovery plan is the business impact analysis (BIA). The BIA is an exercise that combines a current state analysis of core business functions, processes, records, and systems with an examination of various business interruption scenarios. Its purpose is to validate and measure the risks associated with an organization’s core systems, processes, and functions. It provides visibility into the heart of a business to allow executives to make better decisions to enable business continuity during a crisis situation.

The BIA process should result in a document that enables members of an organization to fully understand the priority of functions, with the assumption that in a crisis, limited resources may require a “pecking order.”

Business functions, processes, and systems have differing recovery priorities. Best practices in disaster-recovery planning assign a weighted score to these based on the following attributes, with each attribute identifying a range of scores.

Once the BIA weight has been calculated for each core system, process, and business function, the organization should address the risk items in the prioritized order based on this weight. One of the goals in disaster recovery planning is to lower the BIA weight by influencing the attributes described above.

Step 3: Develop an action plan to mitigate risks.

Risks are always uncertainties, and no amount of planning can eliminate risks. However, a review of the identified threats may often determine that actions can be taken to reduce the probability of a disaster event’s occurrence, or its impact once it has occurred.

For example, if a redundant database system exists, the likelihood of a catastrophic failure is significantly reduced. Security cameras and alarm systems in a facility can alert personnel to dangerous situations before they reach criticality. A well-trained staff can resolve problems more quickly, thereby reducing the duration of the event.

Sometimes it is too costly or resource-intensive to adequately prevent a risk from occurring, but an insurance policy might reasonably cover the costs associated with restoring the functionality of an organization. With careful planning and informed decisions, an organization can reduce the overall BIA weight for most functions.

Step 4: Establish recovery-time and recovery-point objectives.

The next step is to validate the threats from the BIA against their respective recovery-time objectives (RTOs) and recovery-point objectives (RPOs).

The RTO is the maximum amount of time that may elapse before a critical system, process, or function must return to operational state. Each vital system, process, or function will have an RTO that is defined in the context of all others.

For example, long-range patient scheduling systems might be able to be offline for only four hours without causing serious business interruption, while accounting functions may be able to use alternate processes that would allow many days of sustained downtime to standard systems.

A typical range of RTOs might include zero downtime, four-hour recovery, same-day recovery, two- to three-day recovery, four-day to two-week recovery, and one-month recovery. Each organization will place different priorities on the RTO for various systems and functions.

The RPO is the point in time at which a system must be restored prior to moving forward with new data or the next steps in a process. For example, a claims processing system that handles dozens of transactions per second must be able to recover data to the exact point in time when the system crashed. A data warehouse that is updated weekly has a more flexible RPO.

The RTO and RPO are the two key drivers for the entire disaster recovery plan. If a pharmacy determines that it cannot survive more than four hours without its prescription dispensing system, the disaster recovery solution for that system must enable the pharmacy to meet this business objective. This is likely to mean redundant systems and potentially off-site components, all of which will require investment.

Step 5: Write the plan, test it, and review it yearly.

Developing the planning document is a time-consuming but necessary task. The goal is to create a road map that can be followed with minimal decision making at the time of crisis -- because in a worst-case scenario, the functional or technical experts may not be available to assist with recovery. The plan should be structured to allow easy identification of recovery or alternate procedures, along with requirements for certain skills or competencies.

Finally, remember that no disaster plan is ever complete. Organizations change, and environments evolve over time. The plan and associated documentation should be tested annually to verify its level of accuracy and ability to meet the objectives of the organization.

Scott Owens provides project management and business consulting services to health care organizations, helping them meet their business and technology objectives. A certified Project Management Professional (PMP), Scott specializes in the identification and management of initiatives that will reduce business risk and add value through more efficient processes and controls. Scott can be reached at Wipfli’s Milwaukee office at (414) 431-9389 or by e-mail at sowens@wipfli.com.