The vast majority of outstanding repairs have been completed from the original incident period of 10/30-11/2. Over the last two weeks, we have since successfully migrated to a new service to process and aggregate analytics. This upgraded service has resulted in significant improvements to the reliability, resiliency, and performance of our data pipeline systems as we move forward.
Posted Nov 13, 2018 - 11:48 EST
Update
We have been conducting steady repairs over the course of this week. We anticipate our repair efforts will conclude over this weekend, and will provide an update on Monday should any outstanding repairs remain.
Posted Nov 09, 2018 - 15:29 EST
Monitoring
We continue to process & aggregate analytics in real-time, and are monitoring all systems. Users may continue to see inconsistencies in segmentation & flow filtering when relying on integration and delivery event data from 10/30 - 11/2. Outstanding repairs are ongoing.
Posted Nov 05, 2018 - 09:59 EST
Update
We continue to process & aggregate analytics in real-time. There remain periods of time between 10/30 @ 2:30PM EST and 11/2 @ 5PM EST where users may see discrepancies in campaign and flow performance reports. Segments relying on data from this period remain impacted. Flow Filters relying on integration and email events between 10/30 @ 2:30PM EST and 11/1 @ 3:30PM EST may continue to not evaluate as expected. No data has been lost, and we continue to work around the clock on repairs.
Posted Nov 02, 2018 - 17:57 EDT
Update
While we have experienced intermittent analytics backups throughout the day, we are currently caught up and continue to process and aggregate analytics in real-time.
Known Impact: We expect most Flow Filters based on real-time events after 10AM EST today to evaluate reliably and without issue, however some Flow Filters relying on processed events between 2:30PM on 10/30 through 10AM on 11/1 may not evaluate accurately. Segments relying on data from this affected timeframe will also be impacted. We are actively working as quickly as possible to repair outstanding discrepancies around aggregated analytics, and we will provide an update when that repair has completed.
Posted Nov 01, 2018 - 15:00 EDT
Update
We have resolved the analytics backup and are currently aggregating analytics in real-time. The process to repair analytics from the original incident on 10/30 is ongoing, and we will provide an update when that repair has completed.
Posted Nov 01, 2018 - 09:42 EDT
Identified
We are currently back on track with our repairs to address outstanding analytics discrepancies. We anticipate resolving outstanding inaccuracies and backups in analytics processing within the next several hours, and will continue to monitor overnight. We will provide an update in the morning.
Posted Oct 31, 2018 - 21:49 EDT
Investigating
Attempts to repair outstanding analytics discrepancies resulted in a secondary issue that we are currently actively investigating. Segmentation, flow filtering, and report generation are currently degraded. We will provide an update within the next 30 minutes.
Posted Oct 31, 2018 - 17:51 EDT
Update
We are preparing to initiate a large-scale repair to resolve discrepancies in analytics aggregation, and we will provide an update when this repair has completed.
Posted Oct 31, 2018 - 10:45 EDT
Identified
We have identified and released a fix for an issue with our analytics aggregation system. While events were created and correctly recorded, there is a gap in aggregation between 2:30 p.m. on 10/30 through 12:00 a.m. on 10/31. Flow sending, segmentation, and reporting based on events created in that time period are affected, but no data has been lost. We are working to repair the affected data, and will provide another update in the morning.