Post incident report:

Start: 2-March-2023 2100 UTC
End: 3-March-2023 0140 UTC
Duration: 4 hour 40 min

Summary:

Outgoing mail to one region experienced delays due to a configuration error.

Root Cause:

On March 2nd, a configuration change was not successfully applied to one region, leading to delayed delivery of outgoing mail passing through that region.

Mitigation Action:

We resolved the configuration issue in the affected region, enabling mail to be delivered as expected.

Customer Impact:

On March 2nd, some of our customers experienced a delay of approximately five hours in receiving their email.

Follow-up Items and Preventative Measures:

To prevent similar incidents in the future, we are taking the following actions:

• Increasing and fine-tuning monitoring on the outgoing servers to quickly detect and alert our teams to any issues with outgoing mail.
• Improving the automation of the configuration change processes to minimize the risk of manual errors.