Post incident report: Inky deferred incoming mail in Inky legacy.

Start: 29-July-2020 19:00 UTC

End: 29-July-2020 23:00 UTC

Duration: 4 hours

Summary:
At 19:00 UTC, A subset of Inky production services began deferring email being sent to Inky for analysis. AT 19:06 UTC, Inky engineers identified unusual latency in the analysis process specific to a subset of customers. At 20:59 UTC, additional processing capacity was added, which mitigated the impact of the latency. Engineers continued to mitigate and ensure normal mail flow was fully restored by 23:00 UTC. Email previously deferred by Inky systems was then processed as expected.

Root Cause:
Email message processing was slowed due to a processing delay in messages that triggered inappropriate response in the Inky processing system. This resulted in a drop-in capacity resulting in delays of inky mail processing.

Customer Impact:
Customer deferrals began at 19:00 UTC.

Mail was delayed as Inky processing systems deferred messages to customers until production capacity was increased at 20:59 UTC. Following the increase of capacity, deferred email was processed upon resubmission by customer email systems with full function returned by 23:00 UTC.

Mitigation Action:
Inky engineers increased capacity and neutralized the message type that had caused processing delays.

Follow-up Items and Preventative Measures:
• Inky has adjusted preprocessing configurations to identify messages that may cause processing issues and properly avoid impact.. -completed 29-July-2020
• Inky recommends migrating to Inky’s new production infrastructure.