Microsoft O365 sending email to quarantine in EU/UK Region
Incident Report for Inky
Postmortem

Post incident report:

Start: 4-January-2023 0900 UTC
End: 4-January-2023 1330 UTC
Duration: 4 hours 30 minutes

Summary:

Microsoft mistakenly flagged a link (protection.eu.inkyphishfence.com/report) in our EU/UK banners as a phish then sent emails with a banner directly to Quarantine. This resulted in most incoming email in the EU/UK regions in O365 going to Quarantine.

Root Cause:

Microsoft began identifying protection.eu.inkyphishfence.com/report as a dangerous phish URL which then resulted in all incoming mail with that banner to be sent directly to Quarantine for O365 customers. The EU/UK Report This Email link (protection.eu.inkyphishfence.com/report) should have been added with the other Inky links on the Microsoft internal Do Not Block List, but due to an error it was not. This oversight was not caught until Microsoft’s automated systems blocked the EU/UK Report This Email link inadvertently.

Mitigation Action:

We have verified that all Inky banner links are accounted for and properly formatted on Microsoft’s internal Do Not Block list.  We have also added monitoring specifically for EU/UK Regions to ensure we can quickly identify any similar events in the future. We want to ensure that even after emails have been successfully handed back over to O365, they are still delivered as expected to the inbox as intended.

Customer Impact:

All incoming mail for O365 users in the EU/UK regions was being sent directly to Quarantine by Microsoft. When Microsoft corrected the issue with the false positive, they also released all emails incorrectly sent to Quarantine. This had the unintended side effect of emails that were manually released by admins earlier appear to be delivered again and replaced the original receive date with the date they were released from Quarantine.

Posted Jan 11, 2023 - 15:32 UTC

Resolved
We have verified through testing overnight that Microsoft is no longer flagging the original report link in our banners. Any remaining customers that have not re-enabled Inky, would be safe to do so now.
Posted Jan 05, 2023 - 15:12 UTC
Monitoring
Microsoft has corrected the issue. We will continue to monitor and return to the standard links once we confirm that the issue cannot reoccur.
Posted Jan 04, 2023 - 22:19 UTC
Update
We continue to work with Microsoft on identifying the issue and working out the best possible solution.
Posted Jan 04, 2023 - 18:49 UTC
Update
Progress continues on researching initial cause and potential permanent fix for false positive issue. Temporary Fix is still in place and functioning as expected.
Posted Jan 04, 2023 - 17:13 UTC
Update
Fix is still holding and continuing to investigate the issue with Microsoft.
Posted Jan 04, 2023 - 16:12 UTC
Update
Our temporary fix is in place and is working adequately to give Microsoft time to investigate the cause of the false positive and potential remedies.
Posted Jan 04, 2023 - 15:04 UTC
Identified
We have identified the issue being the report link in the banners being falsely flagged as a phishing link by O365. We have implemented a fix to change that link as a temporary measure while we continue to work with Microsoft.
Posted Jan 04, 2023 - 13:28 UTC
Update
We continue to investigate the issue with Microsoft to find out what is causing the issue.
Posted Jan 04, 2023 - 12:24 UTC
Investigating
We are currently investigating an issue with O365 is sending all incoming email to quarantine. At this time, we recommend disabling Inky via the Tools site while we investigate what is causing this.
Posted Jan 04, 2023 - 11:58 UTC
This incident affected: Email Processing (Inky EU).