Inky dashboard connection problem

Incident Report for Inky

Postmortem

Post incident report:

Start: 28-February-2022 1830 UTC

End: 28-February-2022 1910 UTC

Duration: 40 min

Summary:

During a configuration change an error was introduced that briefly impacted the ability of some admins to access the Inky Dashboards.

Root Cause:

Made some changes to configurations to accommodate limitations in url length. After the change was made, one of the servers was unable to connect due to a missed configuration. Once the error was corrected, the server connected as expected.

Customer Impact:

Some Inky Admin dashboards were inaccessible for a small period of time.

Mitigation Action:

Corrected misconfiguration and restarted server.

Follow-up Items and Preventative Measures:

Double check configurations and ensure all servers are accounted for during rollouts
Ensure servers with issues are quickly taken out of the loop of available servers by temporarily removing it from the pool available to the load balancer

Posted Mar 04, 2022 - 14:49 UTC

Resolved

Dashboard issue resolved after configuration change and cycling of servers. Dashboard have been running without connection issues for more than 6 hours.

Posted Mar 01, 2022 - 02:34 UTC

Monitoring

Inky Engineers are monitoring and error rates have returned to normal.

Posted Feb 28, 2022 - 19:10 UTC

Identified

Inky engineers are seeing the connections to back end systems returning to normal. We are monitoring and ensuring that connections return to normal.

Posted Feb 28, 2022 - 18:38 UTC

Investigating

Inky engineers are investigating an issue affecting access to the the Inky admin dashboards. We will update as soon as more information is available

Posted Feb 28, 2022 - 18:30 UTC

This incident affected: Dashboard Services (Dashboard Services US).