Based on the information that has been provided to us by AWS at this point, we believe we have processed all available data from the affected period on July 30th. If you have any questions about data within your account for that period, please reach out to podcast-support@prx.org.
Posted Aug 14, 2024 - 12:41 EDT
Update
All of our systems and AWS's systems have now been operating normally for several days. We continue to work with AWS to understand the full extent of any impact to data that originated specifically during the 22:00 and 23:00 UTC hours on 30 July. We have no evidence at this time that any other data would have been affected.
Posted Aug 06, 2024 - 13:28 EDT
Monitoring
The issue with the underlying AWS service has been resolved. Out of an abundance of caution we are continuing to run our workloads outside of the region that was affected. All data created after 01:43:29 UTC has been processed normally. Our investigation around the data that was delayed during the incident is ongoing, and some of that data will still not be visible in Dovetail Metrics and Dovetail Inventory.
Posted Jul 31, 2024 - 10:09 EDT
Update
AWS has identified the issue and are working on a solution, but the data processing pipeline for Dovetail Metrics and Dovetail Inventory are still experiencing delays. We have taken steps to avoid creating new data in the AWS region that is impacted, so new data should be processed normally. Data (downloads, impressions, etc) that was generated prior to this change may not appear in the apps for some time. AWS has said that a full recovery of their system may occur in the next 1 to 2 hours. Once that is complete, there may be additional time required for PRX to complete our investigation and remediation of the data that was generated during the incident.
We will continue to post updates as they become available, but a complete resolution to this situation may not happen before tomorrow morning. If you have any questions or concerns, please reach out to podcast-support@prx.org.
Posted Jul 30, 2024 - 22:05 EDT
Identified
Due to an ongoing issue with an underlying AWS service, the data processing pipeline for Dovetail Metrics and Dovetail Inventory is experiencing delays. Beginning around 22:40 UTC, downloads, impressions and other numbers in these apps may appear missing or incomplete. There has been no impact to the availability of audio files or downloads to listeners.
Posted Jul 30, 2024 - 20:08 EDT
This incident affected: Dovetail (Analytics Processing, Metrics Dashboard).