Rogers blames massive outage on error during network update
CBC
Rogers Communications Inc. says an update to its network caused a malfunction that "very quickly" took down telecom services earlier this month, according to a letter sent to Canada's broadcasting regulator that was released on Friday.
The outage, which started early on July 8 and for some customers lingered for days, left millions without cellphone and internet service — prompting questions from the federal government and the Canadian Radio-television and Telecommunications Commission (CRTC).
"An update in our core IP [internet protocol] network … caused our IP routing network to malfunction," the letter read.
The letter, posted on the CRTC website, met the regulator's deadline for Rogers to answer questions about the outage. However it has many redactions where Rogers is believed to have offered more specific details about the problem and its plans to prevent something similar from occurring again.
The CRTC said Rogers submitted two versions of the letter, one unabridged and the other with redactions, and that it released the latter to protect "highly sensitive information" about Rogers' operations.
Among other things, the CRTC had demanded Rogers explain why 911 services went down in some areas, and how it plans to honour CEO Tony Staffieri's promise to proactively credit customers' accounts.
"In order to regain the trust of Canadians, it is important that we provide open answers to the questions that they have about the outage," the Rogers letter read. "That is why when answering the CRTC … Rogers is being as transparent as possible."
Rogers also said it has hired a third party to review and provide insights on what happened.
Officials from Rogers and a slew of other stakeholders are set to appear at a parliamentary committee on Monday in Ottawa to further explain the cause of the outage, and to outline the steps they are taking to make sure it won't happen again.
In its letter, Rogers said coding from the update deleted a routing filter that "allowed for all possible routes to the Internet to pass through the routers," which flooded and overwhelmed the core network, causing it to stop processing internet traffic altogether.
"As a result, the Rogers network lost connectivity to the Internet."
It said many Rogers employees looking to fix the problem were affected and could not connect to the company's IT and network systems. As a result, only those "equipped with emergency SIMs on alternate carriers" could initially triage the outage.
"While every effort was made to prevent and limit the outage, the consequence of the coding change affected the network very quickly," Rogers said.
Run 3 Space | Play Space Running Game
Traffic Jam 3D | Online Racing Game
Duck Hunt | Play Old Classic Game
A New Brunswick RCMP officer testifying at a drug trafficking trial this week was questioned by the defence about theft charges he himself faces in an unrelated case.
At precisely noon, the casket welders, sanders and paint sprayers are turned off and a quiet hum settles over the Magog Caskets factory floor in southeastern Quebec.
Edgar Spence keeps a folded bundle of receipts for meal expenses his family has had to cover out-of-pocket since they were forced out of their home community by wildfires in northern Manitoba.
A man accused in a series of north-central Edmonton arsons is guilty of starting fires that damaged or destroyed several homes in 2020 and 2021, a judge ruled Thursday.
Residents of Ward 8 in Hamilton are likely to be back at the polls once again after city councillors voted Wednesday to hold a byelection there to replace former councillor John-Paul Danko, who was recently elected as a member of parliament.
Work on Windsor's large Banwell Road interchange and corridor construction projects is currently on hold after Butler's gartersnakes were discovered.
Etched on a poster board the size of her body, Mary McPherson held up a pencil-drawn portrait of Ontario Premier Doug Ford.
One of the largest old elm trees in Toronto — possibly in all of southern Ontario — is on its last legs.
It's graduation week for many University of Guelph students, but it's a particularly special day for one 68-year-old whose walk across the convocation stage was 49 years in the making.
Scientists on Prince Edward Island have created a new warning system to help Maritime farmers avoid a costly disease called fusarium head blight, which can decrease grain yields while contaminating the crop with toxins.
An N.W.T. deputy minister says the territorial government is hoping to use federal housing money to clear derelict homes from communities.
WARNING: This article contains graphic details, references sexual assault and may affect those who have experienced sexual violence or know someone impacted by it.