An apology from Tucows, regarding the Enom outage

  • January 19, 2022 by Ry Bealey

Ecobyte partners with Enom to provide reliable domain name registrations and services. Enom's planned downtime on January 15th became a multi-day outage that interrupted multiple crucial service channels globally: email, DNS resolution, and SSL certificate generations, among some other systems. Below is the official apology and acknowledgement note from Enom.

Note that, at this time, Ecobyte maintains a strong partnership with Enom and we will continue to offer some products and services utilizing their infrastructure.

--

Beginning Saturday, January 15, 2022, Enom experienced a series of complications with a planned data center migration that caused significant disruptions for a subset of our customers. 

We sincerely apologize to all of those impacted. We pride ourselves on being a reliable domain registration platform, and this weekend we fell short. We are committed to regaining your trust and to serving you better. 

A full internal audit is underway and an incident report is forthcoming. This will include a summary of events and scope, learnings, and policy and process changes to mitigate future issues. 

In the meantime, we have consolidated an overview and timeline of events to transparently share with our valued community. 

Sincerely,

David Woroch

EVP, Tucows Domain Services

--

Below is a general overview of the incident:

Enom had a large-scale data center migration planned for Saturday, January 15, 2022, from 09:00 EST to 21:00 EST. 

We first notified Enom resellers of this migration via email on December 7, 2021, with reminder emails on January 7 and January 12, 2022. In these communications, resellers were told that they would be unable to register and manage domains, email, and SSL certificates through the Enom Control Panel and API during the 12-hour maintenance period, and that DNS services were expected to resolve normally throughout the entire duration of the migration. 

As the data center migration was underway, we experienced delays in transferring our database, which extended the duration of the work period and the length of time registration and management services were offline. 

Alongside this issue, a bug in the new DNS provisioning system resulted in incomplete DNS records for up to 350,000 domains. 

A handful of customers began reporting issues on Saturday, January 15, 2022 at 14:00 EST, which continued overnight. Enom engineers were actively resolving the reported DNS issues throughout Sunday, January 16, 2022, while a parallel effort was underway to determine the magnitude of the problem. All DNS resolution issues—including MX records—were resolved by January 17, 2022 at 13:45 EST.