This incident has been resolved. Metrics affecting click rate and open rate have been fully processed and should now be accurate.
Posted Nov 11, 2023 - 07:50 EST
Monitoring
APIs, Campaigns, Segments, and Forms are now fully operational. Metrics that affect click rate and open rate are in the process of recovery, and should be fully caught up within 30 hours. We are continuing to monitor the issue and will post updates as relevant.
Posted Nov 09, 2023 - 15:36 EST
Identified
We have addressed the underlying issue causing the backups and errors. The incident window was from 1:45pm EST to 2:45pm EST. APIs and Segments have recovered. Campaigns, Forms, and Metrics are recovering and replaying data, and we will provide an update on normalizing when possible. No signup form submits are lost. We will continue to post updates as we resolve the issue.
Posted Nov 09, 2023 - 15:02 EST
Investigating
Starting 1:45pm (US Eastern) we have seen processing delays across multiple services within our system. The issues are causing errors for API requests, email event processing, and segment creation and updates. It may also result in delayed campaigns, and delays for a subset of signup form submits. We are investigating and working to mitigate the issue as soon as possible.