Post incident report:

Start: 22-November-2022 1400 UTC
End: 22-November-2022 1540 UTC
Duration: 1 hour 40 min

Summary:

Small configuration issue combined with greatly increased mail traffic during the week of Thanksgiving combined to delay the delivery of mail to customers.

Root Cause:

On November 22nd, record traffic volumes exacerbated a previously undiscovered configuration issue tied to the scaling up of our infrastructure to respond to increased traffic. It was scaling, but not fast enough to deal with the rapid influx of traffic.

Mitigation Action:

We corrected the configuration error to enable to infrastructure to scale at a quicker rate and we manually increased the number of servers handling mail flow to catch up with the amount of mail traffic that was being affected.

Customer Impact:

On the morning of November 22nd, many of our customers had a period of approximately two hours where email delivery was delayed.

Follow-up Items and Preventative Measures:

Inky has verified the server infrastructure is capable at scaling up quickly enough to match dramatic traffic increases and tested this throughout the remainder of the week including Black Friday and Cyber Monday. We have also added additional alerting on the scaling process itself to ensure we are able to catch any similar issues as quickly as possible and respond with manual action if needed.