Issue in processing webhooks overnight

Incident Report for Klaviyo

Resolved

Event restoration is complete and Webhooks continue to operate normally.
Posted Oct 17, 2018 - 16:55 EDT

Monitoring

Shopify event data should now be restored for the affected time period for all users. Stripe integration event recovery is in progress. Team is actively monitoring this data restoration.
Posted Oct 17, 2018 - 14:37 EDT

Update

For automated flow emails triggered by integration events, this incident may have caused emails to not get scheduled and sent as expected. While we are backfilling all webhook data now, triggered emails that would be over 4 hours late will not get scheduled to send.
Posted Oct 17, 2018 - 12:03 EDT

Identified

Current webhook reception and processing is now normal. The Shopify webhook data backfill operation is in progress, and all users should expect their data restored by 17:00 EST. BigCommerce customers may notice some missing Cart events. We are still assessing the impact of Aftership, Stripe, Mailchimp, WuFoo, Unbounce, and Eventbrite webhooks; an update will be posted when we have estimates on resolution.
Posted Oct 17, 2018 - 11:51 EDT

Investigating

We experienced an issue in processing webhooks from 12:45am EDT to 9:15a EDT. This means segments and flows that rely on events coming from Shopify, BigCommerce, Aftership, Unbounce, Wufoo, Eventbrite, Actblue may be out of date or delayed. We have solved the root issue and are currently backfilling Shopify data. We are investigating this issue further and will provide updates here shortly.
Posted Oct 17, 2018 - 09:31 EDT