Disaster Recovery is Not Enough

Thanks to IP’s increasing popularity and the things it can enable, the DR model has changed. Before anyone can reach the point of recovering from a natural or man-made disaster, there must already be a smooth-running business continuity scheme, because “everybody puts such mission-critical applications on these networks that the notion that you’re going to stop and recover is out,” said Kevin Curran, senior vice president of Optimum Lightpath, the commercial services arm of cable operator Cablevision Systems.

There have always been government and financial users who demanded — sometimes for legal reasons imposed upon them — always-on connectivity and reliability and were willing to pay the price either to a service provider, an outside DR provider, or even the capital expense of building redundant infrastructure.

IP’s ease of use and the overall sense of impatience that drives today’s Internet users have combined to put new strains on those who deliver, store and protect data.

“The focus is on business continuity more than disaster recovery,” Curran said. “What can you do if you’re a service provider [or] an IT department to make sure that your internal network is resilient and that your service provider’s network is resilient, so you can continue to do business no matter what happens: a weather emergency, or some type of flooding in the building, or a terrorist attack, or something like that?”

Lose the Letter D
You can start by not putting all your resources into a disaster recovery plan, said Al Lounsbury, information networking campaign leader at Nortel.

“I like to lose the letter D, disaster, because it’s not for both disaster and recovery, it’s really about recovery should any event ever happen. It could be anything, not necessarily disaster. Most mistakes are human error,” he said.

The carrier is only partially responsible for recovering what’s lost, said Roberta Witty, research vice president at Gartner, because “once it’s in the company, the company then becomes responsible for delivering the business content around. The heterogeneity of IP is what makes it complex.”

Ubiquitous IP has changed the business climate, said Aaron Farajun, executive vice president of Asigra, a software company that works with service providers to deliver backup as part of wider business continuity.

“I don’t know of a telco yet that’s really delivering business continuity as its widest goal,” Farajun said, adding that this is occurring in a climate where “technology is much easier to use, and people use it to generate more data. People are more reliant on technology.”

Businesses no longer run mainframes and work stations. They run data, content and applications over their own (or service provider-supported networks) throughout campuses and to remote locations, where users store their own versions of the data on powerful desktop and portable devices.

Calling this a distributed architecture is like calling a shark a fish: The description’s accurate but hardly tells the whole story.

“Recovering in a distributed computing environment is more difficult than the old mainframe where everybody was on one box,” Witty said.

This means someone — either the network provider or the user — must maintain operations via redundant infrastructure and/or data storage, while maintaining a plan to recover after the fact in the event that a disaster has stripped local resources back to the Stone Age, which is where a tornado, hurricane, earthquake or bomb can leave them.

“The end ultimately is to get the business back up and running,” Farajun said.

Trailers for Coming Events
In some cases more than data is lost, and that’s when a DR plan becomes the end game for the business plan. AT&T has a globalized DR response that includes 150 technology-loaded trailers strategically warehoused throughout the world ready to be deployed to restore network services for short or long periods.

“We can transport them anywhere around the globe, either by air cargo, over the road tractor trailers, cargo ship or whatever,” said Jerry Shammas, executive director of AT&T business continuity and recovery services. “We have our people trained in HAZMAT and the ability to deal with hazardous conditions, nuclear, radioactive. They have the suits and so forth to go into a facility to restore and recover services.”

While other providers have similar DR procedures, AT&T’s ability to take portable, contained facilities around the globe is “really a good differentiator,” said Counse Broders, research director of network services at Current Analysis. “This is one of those areas that AT&T, because of its size, can devote resources to maintaining, and it fits well with its brand image of a company that’s going the extra mile for customers.”

While that’s a nice concept — going the extra mile and all — new age businesses that have put their data in the IP basket know it’s just the way business is done these days. If the network breaks, the carrier fixes it ASAP. If data is lost, somebody finds it — and it better be faster than ASAP. And if there’s a heartbeat of downtime, somebody has to take the blame and pay the price.

“The telecom’s role is changing because … you can’t do business today without using the Internet. They have to step up and provide recovery of that infrastructure,” Witty said.

Business Continuity
That, for want of a better term, is business continuity.

“About 10 to 12 years ago we began to look to … integrating our business continuity preparedness across the corporation, so that we didn’t have just a disaster recovery plan for an application or mainframe or standalone office, we had a business continuity disaster recovery plan and capability for the network overall,” Shammas said.

It’s a common theme across carriers.

“Business continuity really means looking at the immediate temporary restoration of business functions, so a business can get through a disaster,” said Barry Zipp, executive director of managed business applications at Verizon Business. “Disaster recovery is more along the lines of addressing the complete restoration of a business to its pre-disaster condition. They correlate, but their purpose is a little different.”

Put perhaps too simply, everyone benefits when a carrier recovers from a disaster, but only certain businesses — those with both an interest and a financial stake — get top priority focus when it comes to always-on business continuity and data recovery, and that’s usually because they’ve paid for that special treatment.

Customer-Specific Focus
“To say that we have a customer-specific focus is accurate, but I think we are focused on restoration as a whole, as opposed to a piecemeal process that looks at specific components for specific customers,” said Robert Clark, manager of business continuity and emergency management at Verizon Business. “We look at restoring holistically the pieces of the network that support the architecture that supports the customers rather than looking at specific customers and specific line cards and the individual circuits themselves.”

Zipp took it a step further, noting that Verizon Business does “prioritize data in terms of its critical nature, and we have specified procedures to follow to make sure that access to that critical data, access to those critical platforms where even a little bit of downtime is potentially catastrophic, is there.”

Those, of course, are the heavy ends of the user scale. On the lighter end, residential users — even the growing number with home offices — are given lower priority when it comes to restoring services and sometimes given no priority when it comes to restoring lost data.

“Today there is no SLA for consumers,” Nortel’s Lounsbury said. “Consumers need to be aware of that.”

Carriers and service providers make their best effort to assure that service is continuous or restored in a timely fashion. Of course, as VoIP and its attendant needs for emergency 911 capabilities become more widespread, those efforts will become better than best effort.

“Carriers and cable operators are going to have to re-engineer the networks to ensure dial tone gets to the voice-over-IP MTA in the home and that will drive higher SLAs to consumers,” Lounsbury said. “It’s going to take a voice application to drive that [because] most cable companies and phone companies don’t give consumers that high an SLA.”

IP is changing the model for commercial customers, ranging from big to small-medium businesses, by redefining how data is provided, stored and maintained. At the same time carriers are using SLAs and QoS to differentiate their commercial offerings for both data delivery and storage.

More Distributed Environment
“In every way, we’re a more distributed environment, whether it’s distributed in servers moving from mainframe to distributed servers, or distributed in terms that we used to all report to the same headquarters office and now we don’t. We work from home and satellite offices and travel,” said Laurel Burton, product manager in Qwest’s business protection services group.

Burton pointed out that the WAN, not just the data running across it, has emerged as part of the distributed environment and that has “absolutely changed the DR landscape.”

Where in the past service providers and business customers maintained redundant “hot” and “cold” sites, where switches could be thrown and services restored when a portion or localized piece of the network broke down, these models “aren’t necessarily sophisticated enough to help these customers in the event of a disaster,” she said. “We see much more of the dedicated environment, where they have more than one site and they’re actively replicating and synchronizing between them. The DR industry has to change with the times.”

Virtual Tape Libraries
One big change, network data storage, is already happening in the business space and could be coming soon from a residential provider near you. There has always been — and probably will be for a long time — data storage on tape backups.

“The only reason you back up is to recover,” said Adam Couture, senior analyst at Gartner. “We’re moving more and more to virtual tape libraries, running tape backup software, but the actual target is a disk because disks are getting so much cheaper and they’re perfectly appropriate as backup targets.”

Besides, Curran said, “the notion that every Friday night someone takes home the data tapes and brings them to some kind of data repository … those days are over. Most people are realizing that you can much more securely and effectively mirror transactions using fiber optic infrastructure and Ethernet.”

Backup is spreading to the residential space. Rather than storing family photos, videos and other assorted personal data on a hard drive, memory stick or CD, service providers are inviting consumers to upload to the network, a lot of times hiding or sublimating the caveat that “in the cable operator space, most of these Web sites have no SLA attached to them, so if you file your photos up there and that service crashes, they have no responsibility to restore that,” Lounsbury said.

Panelists at this year’s Society of Cable Telecommunications Engineers’ Emerging Technologies conference pointed to the success of storing and distributing video on demand and the potential for network-based content storage from digital video recorders — if the legal hurdles are surmounted — as solid foundations for storing other consumer data on the network, either as backups or as part of a user-generated content offering that the carrier will make available to others on the network.

Either way, it saves carriers from having to put expensive and sometimes quirky data-storage devices in the consumer’s home and helps consumers remove the risk of storing lifetime memories on a computer hard drive.

“I lost my computer, and I lost four or five years of family pictures,” said Mark Dzuban, vice chairman of Cedar Point Communications. “You try to find a better way to store them. Is it a hard drive? Is it doing your own CD? A lot of folks are saying … you want a better place to store it, and you want to have at least access to back your files up at night.”

Carrier-based data storage demands more stringent business continuity and disaster recovery plans. Family photos, for the residential user, are just as important as airline reservations for the commercial player, and there is just as much angst if those files are lost in a disaster.

The Changing DR
“That is a nouveau approach,” Dzuban said. “We have to harden these networks and improve the assumption that when you need that data it’s going to be there and there will be a network for you to retrieve it from.”

This really comes back full circle to disaster recovery because business continuity and the ability to store and retrieve data are steps toward the end goal of making certain no data is ever lost and that not a moment of uptime goes astray.

“The telecommunications companies have a high-end level of responsibility around delivering their core service,” Witty said.

Today, that includes some combination of business continuity, data backup and disaster recovery that fits a 21st century IP-centric model, Burton said, because “the landscape has changed dramatically [and] our models have to change dramatically, because now we don’t work from that central office: We’re all working remotely at some point in time. We don’t rely on mainframes; we’re now in a distributed server environment which has increased the complexity, the sophistication, the ability to recover hundreds vs. one or two.”