Post incident report:

Start: 8-April-2022 0800 UTC
End: 9-April-2022 0100
UTC Duration: 17 hr.

Summary:

INKY’s link re-writing service is one feature within PhishFence that protects users against malicious links. With this service, when an email is initially delivered, INKY rewrites the original link. If a user clicks on a link, the service decodes the link and performs anti-phishing checks at click-time. If INKY considers the link to be part of a phishing attack, INKY presents the user with an additional warning that the link is malicious.

INKY’s link re-writing service originally used the domain shared.outlook.inky.com, but the length of this domain caused certain browsers, in rare situations, to truncate the link. On March 31, 2022, INKY began to use a different domain, link.inky.net, for the link re-write service without any issues.

On 8 April 2022 at approximately 08:00 UTC, link.inky.net was misidentified by Microsoft as a phishing domain. New messages containing URLs rewritten by INKY were blocked by Microsoft phishing protection systems, with those mails being diverted to Office 365 admin quarantine. In addition, previous messages in a user's inbox containing rewritten links were also removed from the inbox and put into the Office 365 admin quarantine by Microsoft’s ZAP software.

In response to customer complaints, INKY researched the issue and identified the incorrect blocking and moving by ZAP of link.inky.net, and at 15:00 UTC, INKY changed the link rewriting domain to shared.inky.outlook.com which prevented future emails from being quarantined. During this same period, INKY contacted Microsoft, and after escalating the ticket to the Microsoft ZAP team, Microsoft performed a remediation action from approximately 8 April 21:00 UTC to 9 April 01:00 UTC which restored the improperly moved emails to the customer’s inbox, with the additional side effect of having the restoration time set as the date/time stamp of the email.

Root Cause:

At approximately 0800 UTC on April 8th, 2022, Microsoft inadvertently added link.inky.net to an internal phishing blocklist due to a false positive classification.

Mitigation Action:

INKY received customer reports of blocked or moved emails at approximately 12:00 UTC on 8 April and began working with the customers to understand the symptoms and research the issue. Once INKY narrowed down the cause of the blocking and moving, INKY proceeded down two parallel paths:

1. At 1500 UTC, INKY changed the domain of INKY’s Link Rewriting Domain away from link.inky.net to INKY’s secondary domain, shared.outlook.inky.com, to prevent incoming mail from being moved into quarantine. This prevented new incoming emails from being quarantined, but the MS ZAP process was still moving previously received email to quarantine.
2. At the same time, INKY filed reports with Microsoft support. At approximately 21:00 UTC, Microsoft escalated INKY’s support issue to the Microsoft ZAP support team, and shortly thereafter, Microsoft identified that the blocking of the link.inky.net domain was incorrect and was a false positive classification. Shortly thereafter, Microsoft initiated a remediation action to reverse their blocking and moved the emails out of quarantine. At 0100 UTC, April 9th, 2022, Microsoft completed their restoration of mail that was incorrectly quarantined.

The mail that was restored to a user’s inbox through Microsoft’s restoration process received new timestamps using the restoration time. INKY continued to work with the Microsoft ZAP team to identify whether the timestamp of restored email could be reset to the original delivery timestamp, however Microsoft concluded that unfortunately this could not occur.

Customer Impact:

The immediate impact of the incident was the move of the affected emails from the user’s inbox to the admin quarantine, and the blocking (and placing in quarantine) of new emails, where the users does not have access. The subsequent impact of the incident was the work required by the customer IT and security staff to manage the restoration of emails from the Microsoft quarantine.

After 21:00 UTC when Microsoft started their restoration, there was additional impact of the restored mail containing a timestamp matching the restoration time instead of the original message time, and also missing information as to whether the email was replied to, causing these restored emails to appear to users as new or duplicate emails. This caused further user confusion and additional work for customer IT and security teams to manage this situation for their users.

Follow-up Items and Preventative Measures:

1. INKY has improved our communications channels with Microsoft support to include the Microsoft ZAP team and has gained additional insight into the ZAP blacklisting process. Our communication with Microsoft will mitigate future blacklisting issues, and we are maintaining our new contact with the Microsoft ZAP team so that we can both proactively and reactively work with them directly in the future.
2. INKY is working with Microsoft ZAP team to formalize the process for whitelisting link rewrite URLs in their systems. The primary and secondary domains for the link rewriting service, link.inky.net and shared.outlook.inky.com, have been verified to be on Microsoft URL whitelists at this time. INKY processes have been updated to utilize this process with Microsoft if any new domains are under consideration for use in the link rewriting service or banners.
3. Using existing Microsoft support relationships and new contacts at Microsoft ZAP, INKY will research ways to add monitoring or processes around the actions of Microsoft ZAP. INKY has already added new monitoring to detect the specific type of blacklisting that occurred during this incident. INKY will continue to perform ongoing risk analysis and ways to mitigate those risks, especially around the way INKY interacts with Office 365 and other external systems.
4. INKY has improved the process by which updates will be communicated to INKY’s incident status web page and stakeholders and improved the system for answering pending questions from customers.
5. While INKY’s change to the domain used for the rewrite url was a planned update to address url length issues and successfully occurred on March 31, that update was not listed in the “What’s New” section of the INKY dashboard. INKY has established new procedures to broaden the scope of what is included on the “What’s New” section of the dashboard with each update.
6. INKY is examining our installation best practices with regard to possible changes that customers could make to their own Office 365 Tenants to further ensure that safe domains used by INKY are not treated as malicious. INKY will send these recommendations by May 1 and will offer any assistance needed for customers to make these changes.

INKY has always placed paramount importance on maintaining high operational availability and great customer experience, including that of IT and security administrators. Our system uptime metrics have always, and continue to, reflect this objective. Although the specific root cause of this issue was due to third-party actions, we realize the large impact this incident had on our customers, including both email end users as well as IT and security administrators, and that there is more INKY can do to both mitigate as well as assist during such events. INKY is committed to implement the above measures and to continue operational and customer experience excellence.