At 12pm EST on 8/23/22, a deployment introduced increased load on our production database. This caused three downtime incidents between 12pm and 1pm EST:
- 12:12pm - 12:21pm EST - 12:24pm - 12:32pm EST - 12:40pm - 12:49pm EST
During these three incidents, the core API for Deadline Funnel stopped responding to requests, which in turn caused core Deadline Funnel functionality to stop working:
We resolved this issue by reverting the deployment and manually killing the long-running queries that had been introduced and that were causing these issues on the production database.
When those steps were completed, both the database and API returned to a healthy state, and all functionality was restored.
The root cause has been identified and additional updates are being made over the next 24 hours to prevent this from happening again in the future.
We apologize for this issue and the inconvenience it caused. Please contact us at help@deadlinefunnel.com if you have any questions.