2020-06-16T15:40:38.802358+00:00
Incident impacting modusCloud - US

UPDATES: Next update - 09:00 UTC (02:00 PT)

02:00 UTC (19:00 PT)

The status has not changed. AD Sandboxing is still in a limited capacity. Next update will be at 02:00 PT.

Everything else is nominal.

Status remaining
Degraded Performance
2020-06-17T02:33:49.586706+00:00

Update on mail issues on US1, US3 and US4 (modusCloud):

00:00 UTC (17:00 PT)

AD Sandboxing message queue is being processed and then released as it is able. Final changes are still being made to fully correct the problem. AD Sandboxing is operating in a limited capacity. Next update will be at 19:00 PT.

Status remaining
Degraded Performance
2020-06-17T00:16:16.238947+00:00

US1,US3 and US4 mail delays update

Next update @ Next update - 00:00 UTC (17:00 PT)

22:00 UTC (15:00 PT)

AD Sand-boxing are being properly accepted and scanned, but not released. Messages queued are now being delivered and the queue is draining. The queue will be monitored and another update will be posted once it is emptied.

20:00 UTC (13:00 PT)

Issue remains stable. Some customers may experience log updates still behind, but queues have drained. Preliminary investigation results show that the issue appears to have been caused by a corrupt virus definition update. We are investigating reports of messages in AD Sand-boxing being delayed.

18:00 UTC (11:00 PT)

Stable. Queues continue to drain. The issue is still being closely monitored.

17:15 UTC (10:15 PT)

Scanning times have been addressed and message scan times have returned to normal. No messages have been lost and any message queues are draining. The issue will continue to be closely monitored and status will be moved to stable

16:45 UTC (09:45 PT)

Message scan times are longer than expected. Scanning engines are being addressed and investigated.

Status remaining
Degraded Performance
2020-06-16T22:14:49.871762+00:00

US1,US3, US4 mail delays and logging delays

UPDATES: Next update - 22:00 UTC (15:00 PT)

20:00 UTC (13:00 PT)

Issue remains stable. Some customers may experience log updates still behind, but queues have drained. Preliminary investigation results show that the issue appears to have been caused by a corrupt virus definition update. We are investigating reports of messages in AD Sand-boxing being delayed.

18:00 UTC (11:00 PT)

Stable. Queues continue to drain. The issue is still being closely monitored.

17:15 UTC (10:15 PT)

Scanning times have been addressed and message scan times have returned to normal. No messages have been lost and any message queues are draining. The issue will continue to be closely monitored and status will be moved to stable

16:45 UTC (09:45 PT)

Message scan times are longer than expected. Scanning engines are being addressed and investigated.

Status remaining
Degraded Performance
2020-06-16T20:30:01.147909+00:00

There are mail delivery delays and logging delays on the mentionned stacks. Proofpoint is working to correct mail flow delays. There is no mail loss to report. Delays and 'Queued' status are being worked on by Proofpoint engineers to return expected mail flow. Dev, NOC, Ops and Support resources are investigating.

UPDATES: Next update - 17:15 UTC (10:15 PT)

16:45 UTC (09:45 PT)

Message scan times are longer than expected. Scanning engines are being addressed and investigated.

17:15 UTC (10:15 PT)

Scanning times have been addressed and message scan times have returned to normal. No messages have been lost and any message queues are draining. The issue will continue to be closely monitored and status will be moved to stable.

[ Vircom Update: 13h47 EST ] -> our tests show that mail seems to be flowing in a timely fashion but the message log itself is catching up. In other words, you could receive an email quickly but the message log would still show the message as "Queued" instead of "Delivered". The backlog should resorb itself soon.

18:00 UTC (11:00 PT)

Stable. Queues continue to drain. The issue is still being closely monitored.

Status remaining
Degraded Performance
2020-06-16T19:15:47.267007+00:00

There are mail delivery delays and logging delays on the mentionned stacks. Proofpoint is working to correct mail flow delays. There is no mail loss to report. Delays and 'Queued' status are being worked on by Proofpoint engineers to return expected mail flow. Dev, NOC, Ops and Support resources are investigating.

UPDATES: Next update - 17:15 UTC (10:15 PT)

16:45 UTC (09:45 PT)

Message scan times are longer than expected. Scanning engines are being addressed and investigated.

17:15 UTC (10:15 PT)

Scanning times have been addressed and message scan times have returned to normal. No messages have been lost and any message queues are draining. The issue will continue to be closely monitored and status will be moved to stable.

[ Vircom Update: 13h47 EST ] -> our tests show that mail seems to be flowing in a timely fashion but the message log itself is catching up. In other words, you could receive an email quickly but the message log would still show the message as "Queued" instead of "Delivered". The backlog should resorb itself soon.

Status remaining
Degraded Performance
2020-06-16T17:48:27.565635+00:00

There are mail delivery delays and logging delays on the mentionned stacks. Proofpoint is working to correct mail flow delays. There is no mail loss to report. Delays and 'Queued' status are being worked on by Proofpoint engineers to return expected mail flow. Dev, NOC, Ops and Support resources are investigating.

UPDATES: Next update - 17:15 UTC (10:15 PT)

16:45 UTC (09:45 PT)

Message scan times are longer than expected. Scanning engines are being addressed and investigated.

17:15 UTC (10:15 PT)

Scanning times have been addressed and message scan times have returned to normal. No messages have been lost and any message queues are draining. The issue will continue to be closely monitored and status will be moved to stable

Status remaining
Degraded Performance
2020-06-16T17:22:45.654580+00:00

There are mail delivery delays and logging delays on the mentionned stacks. Proofpoint is working to correct mail flow delays. There is no mail loss to report. Delays and 'Queued' status are being worked on by Proofpoint engineers to return expected mail flow. Dev, NOC, Ops and Support resources are investigating.

UPDATES: Next update - 17:15 UTC (10:15 PT)

16:45 UTC (09:45 PT)

Message scan times are longer than expected. Scanning engines are being addressed and investigated.

Status remaining
Degraded Performance
2020-06-16T16:57:06.585745+00:00

Confirmation received from Proofpoint. There are mail delivery delays and logging delays on the mentionned stacks. Proofpoint is working to correct mail flow delays. There is no mail loss to report. Delays and 'Queued' status are being worked on by Proofpoint engineers to return expected mail flow. Dev, NOC, Ops and Support resources are investigating.

More information will be posted as it becomes available! "

Status changed to
Degraded Performance
2020-06-16T15:40:38.802379+00:00
Back to Current Status