Inky inbound mail delivery delays
Incident Report for Inky
Postmortem

Post incident report:

Start: 3-November-2022 1300 UTC
End: 3-November-2022 1400 UTC
Duration: 1 hour

Summary:

Delay in processing incoming mail for Office 365 customers for a short time due to a configuration error introduced in a fix for a previous issue.

Root Cause:

On November 2nd, we applied a fix to correct an increase in false positives for some realtime link analysis. This fix contained a configuration error that decreased our processing capacity which impacted the incoming mail during the typical morning email spike in traffic. This decreased capacity resulted in a slowdown in processing for Office 365 customers. This resulted in a delay for the final delivery of the messages for the morning of November 3rd.

Mitigation Action:

We corrected the configuration error introduced in the previous fix. Shortly after fixing the error, we saw the processing rates return to normal.

Customer Impact:

On the morning of November 3rd, some Office 365 customers had a period of approximately 30 minutes where email delivery was delayed.

Posted Nov 14, 2022 - 18:35 UTC

Resolved
Engineers have verified that the error rates are normal and all systems are functioning as expected.
Posted Nov 03, 2022 - 14:32 UTC
Update
Normal error rates have returned and Inky engineers are insuring everything is properly functioning.
Posted Nov 03, 2022 - 14:00 UTC
Monitoring
Mail processing times are returning to normal. Any mail that was deferred due to the slow down will process normally.
Posted Nov 03, 2022 - 13:47 UTC
Investigating
Inky is investigating a issue resulting in delays in processing messages resulting in an increase in message deferals and possible delivery delays beginning at 13:00 UTC
Posted Nov 03, 2022 - 13:38 UTC
This incident affected: Email Processing (Inky Region 1 - Southeast US, Inky Region 2 - Eastern US, Inky Region 3 - Northwest US).