Background processing server misconfigured
Incident Report for PRX
Resolved
All failed deliveries retried and succeeded, all processing jobs retried.
Posted Jul 10, 2014 - 14:10 EDT
Monitoring
Processing is restored, and all jobs have been retried, but there is a backlog of ~200 jobs to execute. We have ramped up processing to finish this as soon as possible. We will resolve this incident once the backlog is complete.
Posted Jul 10, 2014 - 13:23 EDT
Investigating
Our chef server was non-responsive causing a newly provisioned background processor to use an older configuration on launch. This resulted in many failed jobs, such as deliveries and audio processing.
We have restored the chef server to operation and are retrying all failed jobs.
Posted Jul 10, 2014 - 12:55 EDT