Post incident report:

Start: 6-Aug-2022 0400 UTC
End: 11-Aug-2022 0400 UTC
Duration: 6 days

Summary:

Approximately 1% of INKY O365-hosted customers had outbound mail dropped due to a connector issue between Inky and Microsoft. Starting at 0400 UTC on August 6th, 2022, Microsoft began dropping emails from INKY’s outbound email servers directed to affected tenants, with the error “554 5.6.211 Invalid MIME Content: Single text value size (32784) exceeded allowed maximum (32768) for the 'X-Matching-Connectors' header.” The proximate cause of the failure seems to be a Microsoft change to the O365 routing system causing it to consider directing mail sent to a specific INKY tenant to any INKY customer tenant – and therefore add an X-Matching-Connectors header containing the UUID of all our customer O365 tenants; this header value then exceeded 32KB and caused downstream Microsoft to reject (drop) the mail. (To be clear, INKY does not add this header; the header was never in fact visible to INKY servers.)

To temporarily mitigate the issue, Inky advised impacted customers to disable Inky rule 0 to prevent outbound messages from being routed through Inky and obviating the need for Microsoft to properly route mail from INKY’s outbound servers. Inky also engaged resources at Microsoft to begin investigating the issue and began working on standing up infrastructure to support certificate-based outbound routing for all our O365 customers.

Under certificate-based routing, INKY issues a TLS certificate for each INKY customer and uses this certificate in the TLS connection to Microsoft’s server. This appears to prevent the pathological behavior with X-Matching-Connectors header and therefore should prevent future incidents of this sort even if Microsoft reverts their fix for some reason.

Root Cause:

Inky has not received a root cause from Microsoft for the issue which triggered the X-Matching-Connector header to exceed 32KB. However, Microsoft appears to have made a change early on August 11th that fixes the issue.

Customer Impact:

Affected customer mail was not delivered and was responded to with an NDR (Non-Delivery-Report). Unfortunately, in most cases this NDR itself was not successfully delivered back to the initial sender as they were also subject to the X-Matching-Connectors issue.  This left many customers unaware that their mail was not sending.

Once customers employed the workaround of disabling Rule 0, their mail would send, but banner stripping, encryption, and DLP would no longer function.

Mitigation Action:

Inky worked with impacted customers to disable rule 0, to prevent outbound mail from being routed through Inky for banner removal, encryption, and DLP.

Follow-up Items and Preventative Measures:

Inky has deployed infrastructure to support certificate-based outbound routing and will be working to migrate customers to this new method over the next 7-10 days. We believe certificate-based routing will prevent any future similar incident, regardless of whether Microsoft leaves this week’s fix in place.