Office 365 outbound mail problems
Incident Report for Inky
Postmortem

Post incident report:

Start: 6-Aug-2022 0400 UTC
End: 11-Aug-2022 0400 UTC
Duration: 6 days

Summary:

Approximately 1% of INKY O365-hosted customers had outbound mail dropped due to a connector issue between Inky and Microsoft. Starting at 0400 UTC on August 6th, 2022, Microsoft began dropping emails from INKY’s outbound email servers directed to affected tenants, with the error “554 5.6.211 Invalid MIME Content: Single text value size (32784) exceeded allowed maximum (32768) for the 'X-Matching-Connectors' header.” The proximate cause of the failure seems to be a Microsoft change to the O365 routing system causing it to consider directing mail sent to a specific INKY tenant to any INKY customer tenant – and therefore add an X-Matching-Connectors header containing the UUID of all our customer O365 tenants; this header value then exceeded 32KB and caused downstream Microsoft to reject (drop) the mail. (To be clear, INKY does not add this header; the header was never in fact visible to INKY servers.)

To temporarily mitigate the issue, Inky advised impacted customers to disable Inky rule 0 to prevent outbound messages from being routed through Inky and obviating the need for Microsoft to properly route mail from INKY’s outbound servers. Inky also engaged resources at Microsoft to begin investigating the issue and began working on standing up infrastructure to support certificate-based outbound routing for all our O365 customers.

Under certificate-based routing, INKY issues a TLS certificate for each INKY customer and uses this certificate in the TLS connection to Microsoft’s server. This appears to prevent the pathological behavior with X-Matching-Connectors header and therefore should prevent future incidents of this sort even if Microsoft reverts their fix for some reason.

Root Cause:

Inky has not received a root cause from Microsoft for the issue which triggered the X-Matching-Connector header to exceed 32KB. However, Microsoft appears to have made a change early on August 11th that fixes the issue.

Customer Impact:

Affected customer mail was not delivered and was responded to with an NDR (Non-Delivery-Report). Unfortunately, in most cases this NDR itself was not successfully delivered back to the initial sender as they were also subject to the X-Matching-Connectors issue.  This left many customers unaware that their mail was not sending.

Once customers employed the workaround of disabling Rule 0, their mail would send, but banner stripping, encryption, and DLP would no longer function.

Mitigation Action:

Inky worked with impacted customers to disable rule 0, to prevent outbound mail from being routed through Inky for banner removal, encryption, and DLP.

Follow-up Items and Preventative Measures:

Inky has deployed infrastructure to support certificate-based outbound routing and will be working to migrate customers to this new method over the next 7-10 days. We believe certificate-based routing will prevent any future similar incident, regardless of whether Microsoft leaves this week’s fix in place.

Posted Aug 12, 2022 - 22:03 UTC

Resolved
Mailflow has returned to normal.
Posted Aug 12, 2022 - 22:02 UTC
Update
We continue to see the reduction of errors that signify Office 365 customers are no longer having issues sending emails. We are still moving forward with our implementation of a fix that will prevent a reoccurrence of this issue should a similar event occur with Microsoft again.
Posted Aug 12, 2022 - 14:30 UTC
Monitoring
We are seeing a decline in errors related to sending email. We are continuing to investigate the issue with Microsoft while also continuing work on our solution that would prevent a similar incident from happening again in the future.

Of course please let us know by contacting Inky Support if you are still experiencing any trouble with sending email.
Posted Aug 11, 2022 - 12:40 UTC
Update
Continuing to test the fix and how it will impact all our customers.

Contact Inky Support if you are having any issues with outgoing mail.
Posted Aug 10, 2022 - 19:17 UTC
Update
We are finishing up the testing this morning on the fix.

Please continue to reach out to Inky Support if you are still having issues sending emails.
Posted Aug 10, 2022 - 12:16 UTC
Update
Our engineers are continuing to test the fix.

If you are still having an issue with sending email, please reach out to Inky Support.
Posted Aug 09, 2022 - 20:23 UTC
Update
We are continuing to test the fix to restore full functionality to our affected customers.

As always, please do not hesitate to reach out to Inky Support if you are having issues with sending and outgoing email.
Posted Aug 09, 2022 - 18:29 UTC
Update
Testing still proceeds on the fix for the outbound issue.

Make sure you reach out to Inky Support if you are having any issues with sending mail.
Posted Aug 09, 2022 - 17:09 UTC
Update
We are continuing to test the fix to correct the outbound issue.

As always, please do not hesitate to reach out to Inky Support if you are having issues with sending and outgoing email.
Posted Aug 09, 2022 - 16:19 UTC
Identified
We have isolated the issue and a fix is currently being developed and tested. We will continue to post updates here as we have them.

Do not hesitate to reach out to Inky Support if you are having issues with sending and outgoing email.
Posted Aug 09, 2022 - 15:09 UTC
Update
We are continuing to investigate solutions to this issue affecting a small number of Office 365 customers. If you have any issues with outbound emails, please contact Inky Support.

Please Subscribe to Updates on the top right to be notified immediately of updates as we continue to investigate this issue.
Posted Aug 09, 2022 - 14:12 UTC
Investigating
Inky is investigating an issue affecting a small number of Office 365 customers affecting outbound mail flow. If you have any issues with outbound emails please contact Inky support.
Posted Aug 08, 2022 - 16:12 UTC
This incident affected: Email Processing (Inky Region 1 - Southeast US, Inky Region 2 - Eastern US, Inky Region 3 - Northwest US).