Call recordings inaccessible for playback

Incident
October 30th 2023, 11:46pm GMT

Call recordings inaccessible for playback

Status: closed
Start: October 30th 2023, 7:45pm GMT
End: October 30th 2023, 11:46pm GMT
Duration: 4 hours 1 minute
Affected Components:
Americas Contact Center Analytics Contact Center Login Authentication Services and Administration Portal
Update

October 30th 2023, 7:45pm GMT

October 30th 2023, 7:45pm GMT

8x8 is investigating reports of call recording playback receiving errors in 8x8 Agent and Supervisor Workspace and also in the 8x8 Quality Management Analytics. We will provide updates as they become available.

Update

October 30th 2023, 7:54pm GMT

October 30th 2023, 7:54pm GMT

Our engineers are currently working to resolve the issue as a matter of urgency. We will continue to provide updates as they become available. 

Update

October 30th 2023, 8:15pm GMT

October 30th 2023, 8:15pm GMT

8x8 engineers continue to work towards resolution of the issue. We will continue to provide updates as they become available and we apologize for the inconvenience. 

Monitoring

October 30th 2023, 8:29pm GMT

October 30th 2023, 8:29pm GMT

8x8 has resolved the issue causing a delay in availability, however some recordings are still processing and are not yet available for playback. 8x8 estimates this can some hours, however we will continue to post updates once we have a more accurate determination on the estimated time for all recordings to become available. No recordings have been lost and all will be available once we complete the processing of them for playback. Next update will be at 3:00 PM Pacific (6:00PM Eastern)

Monitoring

October 30th 2023, 10:21pm GMT

October 30th 2023, 10:21pm GMT

8x8 has continued to see a decrease in the number of calls still processing. We expect all calls to be fully processed and available before the start of next business day. We'll provide an update on the processing progress at 8pm Pacific time (11pm Eastern) or as an update becomes available. 

Resolved

October 30th 2023, 11:46pm GMT

October 30th 2023, 11:46pm GMT

8x8 has observed all backlog of recordings being cleared. All past recordings are now available and new recordings are available for playback under normal performance metrics. We are doing a full root cause analysis and we apologize for the inconvenience.