On the 2nd of July 2024 Australian users were unable to access the platform during an outage. This article looks to explain what happened, how it happened, and what we're doing to reduce the likelihood of it happening again.
On 2nd July 2024, our system experienced an outage from 1:38 a.m. UTC to 2:58 a.m. UTC. This incident affected customers whose data is hosted in our Australian data center. During this outage, pages such as the login page, calendar and appointment panel were inaccessible, resulting in the system being temporarily unavailable.
The outage was caused by an issue with the older version of the system. Our engineering team took immediate action to resolve the problem. Initially, we increased the number of servers and made some internal adjustments, but these measures did not resolve the issue immediately.
To restore service, we switched the calendar and appointment panel pages to the newer version of the system, which we had been testing for months prior to this incident. This action resolved the issues for these pages. For the login page, we had to replace the servers, which brought the entire system back to normal.
In the days leading up to the outage, we had taken several preventive measures, such as increasing our server resources and updating some internal libraries. Despite these efforts, the issue still occurred.
Looking ahead, we are dedicating additional resources to investigate and resolve the root cause of the problem. We are also being extra cautious with new updates and will conduct thorough reviews to reduce the possibility of other issues further impacting the system, this is in addition to existing checks and reviews that are performed on every update and change.
We understand the importance of a reliable service and apologize for any inconvenience this outage has caused. We appreciate that Zanda is often at the core of how our customer's businesses operate, so the severity of this sort of incident is not lost on us. Our commitment is to provide you with the best possible service, and we are taking all necessary steps to prevent future incidents. Thank you for your understanding and continued support.