Rogers blames massive outage on error during network update
CBC
Rogers Communications Inc. says an update to its network caused a malfunction that "very quickly" took down telecom services earlier this month, according to a letter sent to Canada's broadcasting regulator that was released on Friday.
The outage, which started early on July 8 and for some customers lingered for days, left millions without cellphone and internet service — prompting questions from the federal government and the Canadian Radio-television and Telecommunications Commission (CRTC).
"An update in our core IP [internet protocol] network … caused our IP routing network to malfunction," the letter read.
The letter, posted on the CRTC website, met the regulator's deadline for Rogers to answer questions about the outage. However it has many redactions where Rogers is believed to have offered more specific details about the problem and its plans to prevent something similar from occurring again.
The CRTC said Rogers submitted two versions of the letter, one unabridged and the other with redactions, and that it released the latter to protect "highly sensitive information" about Rogers' operations.
Among other things, the CRTC had demanded Rogers explain why 911 services went down in some areas, and how it plans to honour CEO Tony Staffieri's promise to proactively credit customers' accounts.
"In order to regain the trust of Canadians, it is important that we provide open answers to the questions that they have about the outage," the Rogers letter read. "That is why when answering the CRTC … Rogers is being as transparent as possible."
Rogers also said it has hired a third party to review and provide insights on what happened.
Officials from Rogers and a slew of other stakeholders are set to appear at a parliamentary committee on Monday in Ottawa to further explain the cause of the outage, and to outline the steps they are taking to make sure it won't happen again.
In its letter, Rogers said coding from the update deleted a routing filter that "allowed for all possible routes to the Internet to pass through the routers," which flooded and overwhelmed the core network, causing it to stop processing internet traffic altogether.
"As a result, the Rogers network lost connectivity to the Internet."
It said many Rogers employees looking to fix the problem were affected and could not connect to the company's IT and network systems. As a result, only those "equipped with emergency SIMs on alternate carriers" could initially triage the outage.
"While every effort was made to prevent and limit the outage, the consequence of the coding change affected the network very quickly," Rogers said.
Run 3 Space | Play Space Running Game
Traffic Jam 3D | Online Racing Game
Duck Hunt | Play Old Classic Game
Two of B.C.'s three Independent MLAs have formed a political party that wants to lower taxes, take away teachers' right to strike, and crack down on so-called mass immigration. The party, called One B.C., also wants an end to what it calls B.C.'s "reconciliation industry," and to see the province allow for private healthcare.
An electric scooter rider was rushed to hospital with life-threatening injuries after a crash on the Prince Edward Viaduct on Thursday night, Toronto police say.
Prince Edward Island is launching a new program that makes money available to buy fields where potato wart has been confirmed in the past.
Israel attacked Iran's capital early Friday, with explosions booming across Tehran as Israel said it targeted the country's nuclear and military sites.
An additional round of lead testing is taking place at two Yellowknife schools this week.
More than a dozen Canadian Sikhs live under active threats on their lives, an advocacy organization is warning.
Alberta's measles surge is so dramatic, the last time case counts were higher Calgary did not have an NHL team, O Canada was not yet the official national anthem and gasoline would set you back 24 cents a litre.
Red-winged blackbirds are dive-bombing residents of one Toronto neighbourhood, swooping down and pecking at their heads, but an expert says there's not too much the locals can do as the birds protect their nests.
The Cree First Nation of Waswanipi has officially broken ground on its hotel, marking a milestone for the northern Quebec community after decades of anticipation.
Former prime minister Jean Chretien says dignitaries attending next week's G7 leaders summit in Alberta should avoid engaging with the "crazy" from U.S. President Donald Trump.
The area burned by wildfires so far this season is the second-largest on record in Canada, according to government data.
Environment Canada has issued a severe thunderstorm watch for all of Prince Edward Island.
The Canadian citizen believed to be on an Air India flight that crashed shortly after takeoff is a dentist from Mississauga, Ont., her family has confirmed.