Post incident report:

Start: 16-September-2023 1400 UTC
End: 16-September-2023 1630 UTC
Duration: 2 hours 30 min

Summary:

Mail was delayed and the sections of the Inky Dashboard were unavailable for approximately two and a half hours on Saturday, September 16th due an expired server certificate.

Root Cause:

Due to human error, a server certificate expired before we were able to replace it with a renewed one.

Mitigation Action:

We renewed the server certificate and increased the capacity to process the backlog as quickly as possible to get traffic back to normal.

Customer Impact:

For approximately two and a half hours, users' email was delayed during the period between the server expiring and us installing the new server certificate. During this period, there were also sections of the Inky Dashboard that were unavailable.

Follow-up Items and Preventative Measures:

We have implemented multiple alerting and monitoring processes to mitigate the danger of missing any future server certificate renewals.

Posted Sep 18, 2023 - 14:44 UTC

Resolved

All systems are back to normal and steady.

Posted Sep 16, 2023 - 18:01 UTC

Monitoring

Fix has been applied on the Dashboard. The backlog of emails have cleared with the additional capacity. We will continue to monitor.

Posted Sep 16, 2023 - 16:55 UTC

Update

Fix is in place for mail flow and we are currently expanding capacity to handle the backlog of email that built up during the outage.

Posted Sep 16, 2023 - 16:30 UTC

Identified

We have identified the issue and implementing the fix now. Will update as soon as it is in place and traffic returns to normal.

Posted Sep 16, 2023 - 16:03 UTC

This incident affected: Dashboard Services (Dashboard Services US, Dashboard Services EU) and Email Processing (Inky Region 1 - Southeast US, Inky Region 2 - Eastern US, Inky Region 3 - Northwest US, Inky EU).