Avoiding an IT Crisis: Understanding and Overcoming the Pitfalls of Replication Systems

As data and applications have become critical to an organization’s ability to operate, companies have invested heavily to protect these assets. One technology designed to protect both data and applications is replication. However, despite millions of dollars invested in them, replication solutions often fail in practice, when they are needed most. This paper analyzes the key points of failure and suggests how to avoid them.

Simply stated, replication involves creating a second copy of data as a means of protecting that data. Replication can be leveraged to ensure high availability of mission-critical business applications. This paper focuses on replication applied to the problem of delivering high availability. This scenario is one of the most challenging and complex uses of replication, but when properly implemented, yields high levels of continuity for mission-critical applications.

When used for high availability, replication is performed between two different locations, rather than within the same location. Using multiple locations reduces the risk associated with facility and regional disasters. As the problem focus is delivering high availability, replication is performed continuously rather than in batch mode. Continuous replication maintains the most consistent copy of data without the gaps that occur with a batch approach.

Replication is sometimes confused with clustering, which involves using two or more storage devices at the same location to simultaneously receive the same data stream. This provides two identical copies of the data and mitigates the risk of a disk failure. It does not address other issues associated with high availability. This paper does not discuss clustering or compare clustering to replication.

Replication Solution Overview
A company has many options for how to protect data. including tape backups, disk mirroring, electronic vaulting, and data replication. Data replication is a powerful alternative to other techniques. When properly installed and managed, data replication provides more than data protection, it enables application availability.

There are many ways to deploy a replication solution to safeguard against a company’s most pressing protection risks. Solutions can be deployed either locally or remotely, synchronous or asynchronous. The multitude of options allows a company to deploy an optimal solution based on its need, but creates complexity both in deployment and manageability. This complexity is the root of many challenges associated with replication and often a catalyst of failure.

Data Protection Versus High Availability
When evaluating replication options, it is important to understand the specific reasons for the solution. Two primary reasons for implementing a replication solution are data protection and high availability.

Data protection is primarily focused on data integrity, ensuring that the data and all changes to that data are captured in a backup copy. Immediate access and continued use of the data are not top priorities. For example, it is better to ensure 100-percent data accuracy with medical records than it is to ensure 100-percent real-time access. Inaccurate prescription data may result in loss of life, while a delay in refilling a prescription will only result in a patient becoming annoyed.

High availability, on the other hand, is concerned with three attributes: data integrity, immediate access, and continued application use. A financial clearinghouse needs to ensure its data is accurate, as trade commitments are legally binding. Furthermore, trades need to execute in a timely fashion or else the clearinghouse incurs financial risks. Satisfying the additional need for data access and continued application use requires continuity-driven replication solutions to provide failover and failback capabilities.

A failover is an action initiated by IT personnel to move a secondary application environment from a backup role to a primary role. Usually a failover occurs when a problem develops that prevents the primary environment from properly functioning. The secondary environment starts providing end users with the services originally provided by the primary environment. The data that was replicated to the secondary environment is now put to use. Failback is similar to failover, but returns control back to the primary environment once it is restored to a proper working state. These capabilities turn out to be far more complex and challenging than basic data replication.

Further complicating the technical challenges associated with failover and failback is that high availability solutions must hedge against regional disasters and outages. This requires the deployment of the high availability replication solution in a geographically remote backup environment, ideally more than 20 miles away. Distance requires the complexity of bandwidth latency and cost constraints, factors that are negligible over a LAN.

Using Replication to Provide High Availability
Understanding replication and its prominent place in delivering high availability for mission-critical applications requires a deeper look at the key elements that enable high availability, particularly across a geographically diverse or complex IT environment.

Data Replication – the set of processes that collect and move data from one source to a second source. Data replication includes a process to watch a data repository for new and changed data, which it then duplicates and places in a replication queue. The replication queue stores and manages the pending data transactions, which are then replicated to the secondary data repository. A process monitors and verifies that the transaction arrived safely at the destination. The backup data provides a source safe from many issues that can affect the primary data source. This component provides sufficient coverage to protect data sources when the data is not supplied to applications that require near continuous uptime.

Failover – the set of processes that switch a secondary data source from its backup role to a primary role. Often, this is tightly integrated with other processes that redirect users to an alternative set of application servers. Depending on the application, further security processes and control logic are required. Assuming that all systems are synchronized, all software versions and revision levels are consistent, the systems have been tested, and personnel are standing by at both data centers, the failover can proceed smoothly. During a smooth failover process, users have near-continuous access to their critical applications with minimal interruption.

Failback – involves returning operational control back to the original system. In many instances, a company’s primary environment is more robust or better situated to deliver superior end user performance, benefiting from greater bandwidth or a concentration of local users. For these performance or security reasons, a systems administrator will seek to failback to the primary system as soon as possible once the primary system has been restored to a fully functional state. Failback is similar in many aspects to failover, but adds additional steps and complexity to account for resynchronizing the two systems. This involves recovering the primary system to a working state and reestablishing data integrity, as the two data sources have become out of sync since the initial failover.

Replication Shortcomings
Just as replication solutions offer promises of risk mitigation, higher availability, and data protection, the attempted use of these solutions often results in frustration and wasted time and money for a number of reasons.

First, installation alone can be a deterrent. Companies frequently have so many issues associated with installation or management of a complex solution that they abort their implementation and “shelve” the replication software.

Once installed, many companies find replication manageable. The bigger challenge, however, is encountered in the event of a large-scale systems failure or disaster when attempts are made to activate replication solutions to maintain application availability and recover. Failover processes are difficult to test and don’t always work as expected.

Accounting for all permutations of failover requires extensive process mapping across the entire IT environment that supports the application. Finding the expertise required to perform this mapping is difficult and rarely exists within one group in a company, or even throughout the entire company. Failure to adequately address all possible failover scenarios creates risk and possibly a false sense of security.

Top 5 Causes of Replication Failure
An examination of the top five causes of replication failure provides insight into the challenges a company faces when seeking to successfully implement a replication solution, as well as a useful checklist for benchmarking purposes.

1. Secondary Environment Not Ready for Failover
A company establishing a secondary environment to provide superior protection of its most important applications and data must address many challenges. One of the biggest challenges is maintaining two nearly identical environments. All software, patches, and access levels need to be consistent. Over time, many changes that are made to one system fail to get implemented on the other system. These differences often result in a secondary environment that is out of sync with the primary environment to the point where failover cannot occur.

Another factor impacting failover readiness in the secondary environment is that critical processes do not function properly. Most organizations do not have clear visibility into the readiness of their systems for failover, and are surprised by one of the following problems:

• Replication is not performing normally
• The replication queue is too large
• The secondary environment is not healthy
• Primary/secondary environment software and configurations are out of sync
• Dependent systems are not designed for failover

It is critical for a company to take the time necessary to develop processes to control the introduction and distribution of changes and updates to both environments. One undetected change can cause hours- or days-long delays in service resumption. It is equally important to monitor all critical processes that impact the readiness of the secondary environment, as early problem detection ensures system readiness when a situation demands use of the secondary system.

2. Manual Error in Failover Process
People are often the weak link in a failover process. Manual errors introduced during a failover sequence can corrupt the entire process. This results in a need to fix the problems resulting from the error and further requires the entire process to be restarted.

People are more prone to make mistakes during a crisis. The more complex and step-intensive the failover process, the more likely mistakes will occur and result in failure. As an example, it takes 350 steps to failover a 10-server Microsoft Exchange environment. Any single human error in that sequence will break the failover process. A few examples of mistakes include:

• Missed process steps
• Steps executed out of sequence
• Process initiated before dependent steps fully executed
• Misjudge state of primary or secondary
• Typing error
• Steps out-of-date for current software versions

Often, these steps must be performed under pressure or over a remote connection. The steps required to remediate errors introduced during a failover are complex, often uncharted, and should be avoided at all cost.

3. Experts Not Available During Crisis
Failover processes are dependent on expert staff who may not be available during a crisis. The failover process touches upon all technical disciplines, from hardware and operating systems, to applications and databases, to networking and security. In a large organization, these disciplines are highly specialized with different personnel responsible for each. If any one person is unavailable, the failover process can break down. There are many reasons people are not available, including:

• Occupied with other crisis efforts
• Physically unable to access facilities
• Can’t be contacted during an emergency
• No internet access available to control failover

As part of the process to develop and deploy a failover solution, it is important to establish a list of required skills and resources available for each skill. Full contact information and predetermined communications protocols need to be created, continuously updated, and readily available to all team members. These steps will aid recovery efforts and mitigate some personnel risks.

4. Failover Process Unable to Scale
In large organizations, the scale of a failover or recovery effort can become a critical bottleneck. The technical staff is limited, often managing large numbers of systems. A critical application failure or a facility problem can result in dozens or more systems that require failover. Multi-server failover is a serial, manual process. The technical staff comes under tremendous pressure to rapidly restore service. Under these conditions, scaling issues are likely, such as:

• One administrator can only failover one server at a time
• A 25+-server environment will take 10+ hours for one administrator to failover or failback
• Many mutually dependent systems will not work until the entire environment has failed over

5. Untested Failover Assumptions Don’t Work
Complex multi-server failover is often too sensitive to fully test. Without testing every permutation of systems and failure causes, it’s impossible to know exactly what will happen during a real crisis. Issues that contribute to failover breakdowns include:

• Large, complex environments can have many failure types and scenarios
• Multi-server failover involves constantly changing conditions
• Server-by-server failover is very different from a holistic failover of the entire environment
• Different failures result in different behaviors

Some methods available to mitigate risk include incorporating “what-if” scenario planning sessions and “pre-mortems,” a form of role playing that allows a technical staff to identify untested failover scenarios and potential bottlenecks.

The Impact of Replication Failure
Understanding the impact of replication failure is essential to making informed business decisions about replication investments. The following provides a summary of the key issues associated with replication failure.

Replication failures have long recovery times. Complex recovery processes take time to implement. Restoring an entire environment requires significant skills. The availability of critical resources and personnel becomes a bottleneck. Often, there are periodic delays during the failover process as key skills are not available. If large amounts of data must be moved as part of a recovery effort, bandwidth constraints can further prolong the recovery time.

Replication failures can cause other problems. Replication failures can have wide-spread impact. Database corruption can occur, requiring substantial efforts and time to restore a database to a functional state. Even when the databases are not corrupted, effort is required to determine how much data was lost during the failure and to recover the missing transactions.

Replication failures have a high cost. Replication is almost exclusively used for critical systems. Data in these systems is important enough to protect and therefore to recover. The time and effort required to recover the data, returning it to a usable state, is substantial. Replication failure forces application downtime, resulting in negative economic consequences: lost revenue, missed opportunities, degraded customer satisfaction, and declines in shareholder value. Any replication failure will prove costly.

To Succeed, Eliminate the Risks
Replication solutions represent a powerful way to protect a company’s most important data and applications. But replication solutions, particularly those that are used to provide failover and failback for high availability environments, are fraught with risks, often resulting in a low success rate for companies attempting to implement these solutions. The key to successfully implementing a replication solution is to understand all the risks and eliminate as many as possible. Planners should start with the risks identified above in the top 5 reasons for failure. Through careful evaluation of approaches, they can select a solution that matches the best approach for the company.