Start: 8-March-2022 1323 UTC
End: 9-March-2022 1630 UTC
Duration: 27 hr 7 min
The Inky Dashboard is having intermittent loading issues for some customers.
Resource usage on the servers was higher than expected. Systems put in place to help in cleaning up the resources used were so busy they started to use more resources than expected. The cycle continued until some systems were failing when users were trying to access the dashboard.
Some users were unable to access the Inky Dashboard intermittently.
Separated each portion of the process out to its own server, so they can scale independently and quickly.
The entire process has been split with each step having its own resource pool to draw from. This should prevent any one step overwhelming the entire process.
Monitoring has also been broken down to quickly spot an issue with any individual step in the process in addition to monitoring the health of the process as a whole. This should assist us with spotting intermittent issues like this one.