On Friday, July 8 beginning at 4:44 am EDT, Rogers Communications, Canada’s largest mobile and internet provider suffered a catastrophic outage.

More than 11 million customers including banks, ATMs, managed service providers, hospitals and emergency service hotlines were impacted.  Netblocks, a UK based organization that monitors cybersecurity, estimates that 25% of Canada’s internet was knocked out and the outage lasted at least 15 hours paralyzing communications across many sectors.

One SMB customer, impacted for 2 days by the outage, asks their managed service provider…

 “We were unable to access our server on Friday and Saturday. And, we have had several odd things happen after the service was restored.  Can you give us an update as to what happened? We know Rogers had an outage. Was this the cause? Do you have a backup telecom carrier?”

MSP Support Reply – 1 week later

 “Our servers are physically located within a Rogers Tier 4 Database. All physical connections and routes are handled by Rogers Communications. On the Friday when you could not connect, Rogers in Canada had a major outage nationwide. This affected many businesses across Canada and even left some emergency services inaccessible (911 calls, debit machines, etc.).

This major outage was out of our control entirely so realistically there was nothing we could do. This is not common for Rogers to have a nationwide outage.”

Customer Wonders:  Was this entirely out of the MSP’s control?

 “When the Rogers outage happened, I assumed that this must be the reason for our service interruption.  At that time we reached out to numerous experts in the business continuity community to ask what should be expected in terms of telecom continuity. I was told that it is common practice for a managed service provider to have a backup telecom service — in case of an outage of the primary telecom provider.  Because your customer base includes banks I “assumed” that telecom redundancy was in place.

Yet, your email said, “This major outage was out of our control entirely so realistically there was nothing we could do.”

Finally, it would have been helpful for your customers to have some sort of communications from your team immediately so we knew what was happening. We didn’t hear anything from your company for over a week.  Ideally the communications would have been an email to your customers and a status update on your website. Could you speak to these issues?

The CEO’s Reply

 “Unfortunately this was bigger than just an outage. 911 services, banks, ATMs and stores were down. I couldn’t even get cash from an ATM. Many banks and stores had to shut their doors. Rogers runs a good 75 percent of Canada infrastructure so that’s why we are all waiting to find out what happened. The backup networks system was down as well so no matter what we had in place it would have been down. Hospitals were offline as well as all home phones, internet and cell phones.  We couldn’t put out any notice because all systems were offline and we were scrambling to get any info in or out.

Rogers is updating us with what happened, but they say it will be some time before we find out the story.

I am sorry about this, but Rogers was to make sure the backup system went online and they failed in that as well. With 911 services offline for over 36 hours, when by law they are to make sure that they never go down, means that something major happened and investigation is under way.

I wish I had more info but as most people; I’m in the dark as well. We will be updating once we have more info.”

Join the discussion on LinkedIn…what is your advice for the MSP? For the customer?

• What should this Managed Service Provider do to improve their future service and ensure business continuity for their clients?
• How should the customer assess their current/future needs and secure reasonable service levels?
• What has your company done to ensure telecom service continuity?

Come to LinkedIn and weigh in on this discussion  >>