The root cause of this incident has been identified and stopped. We are deploying a more robust permanent fix should this ever happen again and will continue to monitor the system for the time being.

It's back, our fixes were not enough. We will apply more robust measures.

This incident was due to abusive behavior from a customer. We have now deployed several different countermeasures and contacted the customer to clarify the behavior and stop it.

We will continue to monitor response times for the time being.

We have deployed a temporary fix and are still working on a more permanent one.

Our cloud provider is continuing to work on a solution. We may need to take the app down for maintenance to get it fixed. In that case we'll accept the longer response times for the day and perform the maintenance during the weekend. Response times are normal right now, but they may increase again depending on load and usage.

All systems are back to normal.

All servers are back online. We are resynchronizing replicas and the app may feel slower until all systems are back in sync.

Our main database is running but a replica is unresponsive. We are working with our cloud provider to fix it.

The app is having trouble creating new connections to our main database, which degrades performance and prevents background jobs such as stat and report generation, imports, pipeline automation, etc from running. We are in contact with our cloud provider.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.