Avoiding the Domino Effect

To a cynic, not just humans but nature, too, might seem to be conspiring to upend the digital economy. With 9-11 not yet a distant memory, the fading of summer witnessed massive power outages in the United States, Canada and Italy, an earthquake in northern Japan, and the fury of Hurricane Isabel along the mid-Atlantic states. There probably are few IT managers who need to be convinced further of the importance of being able to recover data in the event of a disaster.

Replicating data to another site is an important part of an insurance policy for business continuity. Whether critical data is stored at another site within the metro, across state boundaries or even cross-country, transport services are necessary to connect geographically dispersed locations. Little surprise then that network equipment vendors and carriers led primarily by the IXCs and, to a lesser extent, the ILECs, are jumping into the network storage extension space. This is relatively new ground for carriers because until recently the very large enterprises preferred the do-it-yourself approach of deploying metro DWDM systems to support ESCON/ FICON for mainframe connectivity and, in the open systems world, Fibre Channel for SAN (storage area network) extension.

There are basically three ways – WDM, SONET and IP – to extend a SAN, or four, if you consider ATM. But ATM is considered a legacy technology due to high overhead and lack of scalability. Which of the three a customer selects will need to take into consideration factors such as cost, capacity, line rate and, of course, the availability of the preferred transport service. BellSouth and Verizon claim to support all three transport options both within their footprints and also out of region using either their long-distance service or partnering with an IXC. However, depending on a customer’s distance requirements, IXCs might possibly have an advantage here because of the larger reach of their networks.

Government regulations from agencies such as the SEC, Department of Homeland Security and the Department of Health and Human Services have set new regulations for business continuance and disaster recovery. At the same time, near catastrophic events underscore the need for storing data in geographically dispersed sites. How important the availability of the data is for a customer will affect the choice of whether the data is replicated asynchronously or synchronously to a back-up location (see “The Replication Game”).

“The more automated a business becomes, less is the tolerance for downtime,” notes Paul Digiacomo, business continuity management director for AT&T. On the other hand, business continuity does not imply that a customer have access to all of the data all of the time, but access only to the most missioncritical applications. While the financial services industry has taken the lead in protecting information, other segments, such as just-in-time inventory management and CRM applications, might also require data replication. “For those firms that grew up on the web, it’s not uncommon to have dual [data centers], because if you lose your web presence, your store is down,” says Digiacomo.

Network Storage

The demand for storage is escalating as businesses generate increasing amounts of information. The Yankee Group estimates that the network storage market will grow from $14.34 billion this year to $24.23 billion in 2005. The term network storage encompasses two types of schemes: NAS (network-attached storage) and SAN (see “Defining Network Storage”).

The challenge for equipment vendors and carriers is to offer a Fiber Channel extension capability that can extend the protocol beyond approximately 10 km – or about 8 km for ESCON and 100 km for FICON. To compensate for Fibre Channel’s flow control mechanism and ensure that there is no loss of Fibre Channel frames, the industry has developed a buffer-to-buffer credit management system for throttling back the transmission of frames so end stations can confirm the successful transfer of dataa concept analogous to the TCP window. Thus the number of buffer credits needed to, say, support 2-Gbps Fibre Channel will be more than those needed to support 1-Gbps Fibre Channel over the same distance. Additionally, the smaller the frame size, the greater the number of buffer credits.

“The problem [of transporting Fibre Channel] over a long distance is that the roundtrip delay limits throughput because it limits the number of outstanding credits you can have,” notes Jacob Larsen, business development director for Lucent’s optical networking group. The buffer credit hurdle, however, can be circumvented by allowing for a greater number of outstanding credits or through spoofing. This issue would seem to give IXCs the advantage over ILECs in transporting Fibre Channel over long distances as they’d be able to do end-to-end tuning of the buffers. Buffer management aside, asynchronous data replication will be more tolerant of greater distances between SANs than synchronous replication.

Choices
Currently, DWDM seems to be the lead technology for SAN extension in the metro. In addition to offering high fiber capacity, DWDM supports transparent transport of SAN protocols at line rates as well as other services such as GigE, voice and video. Multiple Fibre Channel connections as well as ESCON/FICON services can be aggregated on a single wavelength. SONET, like DWDM, while also supporting full line rates, though offering less capacity, can transport storage protocols transparently. Some systems also do buffer credit management to enable greater reach for Fibre Channel. AT&T, for example, claims to handle Fibre Channel over DWDM across a distance of about 200 km and approximately 100 km for Fibre Channel over SONET for synchronous replication of data.

“The question we get asked frequently is whether storage over WDM competes with storage over SONET. We’ve done a lot of economic modeling, and the two technologies are not competing. If you need the equivalent of an OC48, DWDM is cheaper. The price of DWDM is very cost effective in the metro today. But if you need less [than an OC48] or even fractions of that, you can do very well with SONET,” says Larsen.

Due to the popularity of IP and Ethernet, the interesting development has been the transport of Fibre Channel over IP, or FCIP, as it’s called. An alternative is to do away with Fibre Channel entirely and transport SCSI over IP (iSCSI). Earlier in the year, Cisco released a case study on FCIP detailing the transport of Fibre Channel across Sprint’s SONET network between the carrier’s Burlingame, Calif., and Kansas City, Mo., sites. In the trial, a Hitachi storage array interfaced via a 2-Gbps Fibre Channel to a Cisco Fibre Channel switch supporting an IP services module. The Fibre Channel switch, in turn, had a GigE IP link to a Cisco Catalyst switch. The latter switch had an OC-3 POS (packet over SONET) interface to a Tellabs DACS (digital crossconnect system), which encapsulated the OC-3 circuit within an OC-12 as an STS-3c. The same configuration was repeated at the other end of the OC-12 connection. The trial was used to replicate a large Oracle database using Hitachi’s asynchronous replication technology.

Next-gen SONET platforms or MSPPs (multi-service provisioning platforms), which are fast gaining popularity with established IXCs and, more recently, with ILECs, offer a way to support FCIP using either Ethernet over SONET or (if a router is inserted between an FCIP-capable Fibre Channel switch and the MSPP) running FCIP across a SONET network, as in the Sprint trial. Some MSPP vendors are also starting to support Fibre Channel interfaces on their equipment, enabling carriers to offer a managed Fibre Channel-over-SONET service. With BellSouth dual-sourcing MSPPs from Cisco and Lucent, Mark Kaish, BellSouth’s vice president of data product management, says that, through BellSouth’s longdistance arm, the company is working to offer Fibre Channel-over-SONET services to a few customers. Kaish is also bullish about the prospects of FCIP over Ethernet. “It’s where the power of the metro Ethernet comes in. [FCIP] is clearly a growth market, and two or three years from now it’ll be like IP telephony or any other technology that gets democratized and, to some extent, commoditized.”

Synchronous disk mirroring may be out of reach of all but the very large enterprises. The opportunity, instead, may be in asynchronous replication of data for large and even smaller business that need real-time access to data in the event of failure. And though FCIP might offer the opportunity of bringing SAN extension services for asynchronous replication to the approximately 95 percent of SMBs (businesses with less than 100 employees), not everyone is so sure that this market is ready to be tapped. “The SMBs have still to get into the SAN market as the SMB space is dominated by NAS technology,” says Ed Chapman, senior director of product management for Cisco’s storage technology group. “It’s a tough call to say whether FCIP will go down market because what it really lets you do is take advantage of longer transmission distances.”

Jack Hunt, marketing director of optical storage connectivity for Nortel, believes that FCIP is pretty much a lowend play. “Packet loss in the public Internet is about 5 percent, so no one is thinking about SAN extension over the Internet. But even in a managed IP service with a typical SLA, the packet loss is 0.1 percent. So if you take this kind of packet loss, the maximum throughput will be 29 percent at most, even if you were paying for 100 Mbps-200 Mbps of storage connectivity between New York and Chicago,” Hunt says.

According to Hunt, rather than use a POS interface on the router for FCIP, it makes more sense to put Fibre Channel traffic directly over SONET by taking advantage of an MSPP’s GFP (generic framing procedure) for mapping Fibre Channel to SONET and its virtual concatenation feature for making more granular use of SONET bandwidth. “With IP, the only choices that an enterprise has with routers is a [POS] OC-3, OC-12 or OC-48 interface. Our studies show that [Fibre Channel directly on SONET] is far cheaper than doing storage over IP. So that would be the way to jump-start the SAN extension market for smaller companies,” Hunt says.

Take Off?
Over time, service providers and their customers together will sort out the SAN extension issues. Led by the financial services industry and to a lesser extent by the health care and government segments, SAN extension is clearly a high-end proposition for now. Smaller customers are expected to continue to aggressively adopt NAS as their preferred storage solution because it calls for an IP extension as opposed to an expensive SAN extension infrastructure. However, Lois Horenstein, director of business development for Verizon’s enterprise solutions group, which claims to have about 10,000 enterprise customers, says Verizon sees a demand for SAN extension beyond the Fortune 2000 companies. Horenstein noted that in September, the carrier announced that Amscot Financial of Tampa, Fla., would use Verizon’s supplied EMC storage solutions to replicate Amscot’s data to an off-site data center.

Network storage vendors such as EMC are tapping into a growing market not just by offering large refrigerator – size solutions that can hold more than 80 terabyes of data – and even petabytes, when networked – but also smaller storage devices priced from about $10,000. Working in partnership with network equipment vendors such as Cisco and Nortel, and carriers such as Verizon, which resells EMC products, these companies are pushing to broaden the SAN extension space. Eventually, service providers and their customers will work out the right choices for disaster recovery. At this time, AT&T claims to offer the broadest portfolio of SAN extension services, including supporting asynchronous replication worldwide. “Tell us what interface you want, and our network engineers will make the choice of the underlying technologies,” says Digiacomo. AT&T can expect more competition because for customers, who put a premium on business continuance, hours or days of downtime is not an option. Moreover, recent events have shown it’s not enough to have a back-up facility several hundred miles away, but thousands of miles away. And the time to start thinking about this has already come.