2020-08-11T17:23:04.629660+00:00
Incident impacting modusCloud - US

UPDATES:

[Vircom: Currently, things are back to Normal.]

[Proofpoint: August 12th 13:30 UTC (06:30 PT)]

Please be aware that, between 16:51 UTC (09:51 PST) and 17:45 UTC (10:45 PST) on August 11th, users may have noticed that an excessive number of messages may have sent to quarantine erroneously. Administrators and end users receiving digests can manually release messages as required. Standard checks will be applied to messages upon release from the quarantine. Product Engineering are investigating a bulk release mechanism across the platform.

Status changed to
Fully Operational
2020-08-12T15:27:04.792969+00:00

17h50 EST: So far, there has not been any new update on the Proofpoint side however things seem to be back to normal for everybody. We're still trying to push Proofpoint to do a bulk release for outbound email caught by this issue. I will be monitoring updates and news from our contacts at Proofpoint during the evening here. If I hear something new, I will let you all know.

- Yves Lacombe, Director of Support, Vircom

Status remaining
Degraded Performance
2020-08-11T21:52:11.002329+00:00

DESCRIPTION

SERVICE(S): Mail delivery

DATACENTER(S): All Servers

START TIME: 11 July 2020, 16:51 UTC (9:51 PT)

END TIME: 11 July 2020, 17:29 UTC (10:29 PT)

STATUS: STABLE

DETAILS: All outbound messages are being quarantined due to elevated spam scoring. Proofpoint is working to correct mail flow delays

UPDATE:

next update in 60 minutes: 21:00 UTC (14:00 PT)

20:00 UTC (13:00 PT)

The release of affected messages is still being investigated.

19:16 UTC (12:16 PT)

Upon further investigation some customer's inbound messages may have been impacted.

19:00 UTC (12:00 PT)

Monitoring. Recovery is still being investigated.

18:30 UTC (11:30 PT)

Spam definitions blocked messages with hyperlinks that included HTTP. Cleanup methods are being explored to return mail that was caught by this definition.

18:08 UTC (11:08 PT)

Engineering believes the issue has been corrected, but customers are still being impacted. Root cause is still being investigated.

18:00 UTC (11:00 PT)

Issue is still being investigated. Initial trouble-shooting points towards a bad Spam definition

Status remaining
Degraded Performance
2020-08-11T20:03:26.698328+00:00

DESCRIPTION

SERVICE(S): Mail delivery

DATACENTER(S): All Servers

START TIME: 11 July 2020, 16:51 UTC (9:51 PT)

END TIME: 11 July 2020, 17:29 UTC (10:29 PT)

STATUS: STABLE

DETAILS: All outbound messages are being quarantined due to elevated spam scoring. Proofpoint is working to correct mail flow delays

UPDATE:

next update in 60 minutes: 20:00 UTC (13:00 PT)

19:16 UTC (12:16 PT)

Upon further investigation some customer's inbound messages may have been impacted.

19:00 UTC (12:00 PT)

Monitoring. Recovery is still being investigated.

18:30 UTC (11:30 PT)

Spam definitions blocked messages with hyperlinks that included HTTPS. Cleanup methods are being explored to return mail that was caught by this definition.

18:08 UTC (11:08 PT)

Engineering believes the issue has been corrected, but customers are still being impacted. Root cause is still being investigated.

18:00 UTC (11:00 PT)

Issue is still being investigated. Initial trouble-shooting points towards a bad Spam definition

*******

Note from Vircom:

Individual users with emails in their quarantine complaining -> release their mail

Individual customers with time-sensitive mailflow -> release their mail

All other customers -> wait for PP to mass release

We're pushing PP to do a bulk release of mail.

As for the inbound issue detected, this probably affects a small scope of clients. The inbound mail impacted in the latest update is probably caused by some orgs that use something like Exclaimer with office365 or people who'se domain are hosted on googlel ... inter-user emails in the same domain, instead of staying on the MTA is actually routed via MX resolution so hits proofpoint on the inbound. So if you're not using exclaimer or not on google apps ... we suspect you are okay (but we're waiting for more info from PP)

Status remaining
Degraded Performance
2020-08-11T19:33:45.019911+00:00

DESCRIPTION

SERVICE(S): Mail delivery

DATACENTER(S): All Servers

START TIME: 11 July 2020, 16:51 UTC (9:51 PT)

END TIME: 11 July 2020, 17:29 UTC (10:29 PT)

STATUS: STABLE

DETAILS: All outbound messages are being quarantined due to elevated spam scoring. Proofpoint is working to correct mail flow delays

UPDATE:

next update in 60 minutes: 20:00 UTC (13:00 PT)

19:00 UTC (12:00 PT)

Monitoring. Recovery is still being investigated.

18:30 UTC (11:30 PT)

Spam definitions blocked messages with hyperlinks that included HTTPS. Cleanup methods are being explored to return mail that was caught by this definition.

18:08 UTC (11:08 PT)

Engineering believes the issue has been corrected, but customers are still being impacted. Root cause is still being investigated.

18:00 UTC (11:00 PT)

Issue is still being investigated. Initial trouble-shooting points towards a bad Spam definition

*****

Note: Vircom is pushing Proofpoint to do bulk release of messages that were caught during the outage period however you can release them yourself if need be. Again, thank you for your patience to the thousands of organizations under us.

Status changed to
Degraded Performance
2020-08-11T19:13:51.541464+00:00

Here's the latest update from proofpoint:

DESCRIPTION

SERVICE(S): Mail delivery

DATACENTER(S): All Servers

START TIME: 11 July 2020, 16:51 UTC (9:51 PT)

END TIME: 11 July 2020, 17:29 UTC (10:29 PT)

STATUS: RECOVERING

DETAILS: All outbound messages are being quarantined due to elevated spam scoring. Proofpoint is working to correct mail flow delays

UPDATE:

next update in 30 minutes: 19:00 UTC (12:00 PT)

18:30 UTC (11:30 PT)

Spam definitions blocked hyperlinks that did not have "https" included. Cleanup methods are being explored to return mail that was caught by this definition.

18:08 UTC (11:08 PT)

Engineering believes the issue has been corrected, but customers are still being impacted. Root cause is still being investigated.

18:00 UTC (11:00 PT)

Issue is still being investigated. Initial trouble-shooting points towards a bad Spam definition

Note that you should be able to release messages manually that were caught outbound, however we are in discussion with their ops team to see if they could do a bulk release for all tenants (not including vircom) ... so as soon as we have news on that we'll let you know.

Thank you for your patience.

Status remaining
Major Outage
2020-08-11T18:47:28.024849+00:00

DESCRIPTION

SERVICE(S): Mail delivery

DATACENTER(S): All Servers

START TIME: 11 July 2020, 16:51 UTC (9:51 PT)

END TIME:

STATUS: INVESTIGATING

DETAILS: All outbound messages are being quarantined due to elevated spam scoring. Proofpoint is working to correct mail flow delays

UPDATE:

next update in 30 minutes: 18:30 UTC (11:30 PT)

18:08 UTC (11:08 PT)

Engineering believes the issue has been corrected, but customers are still being impacted. Root cause is still being investigated.

18:00 UTC (11:00 PT)

Issue is still being investigated. Initial trouble-shooting points towards a bad Spam definition

Status remaining
Major Outage
2020-08-11T18:13:04.702279+00:00

SERVICES: Mail Delivery

DATACENTER: All servers

START TIME: 11 July 2020, 17:11 UTC (10:11 PT)

STATUS: Investigating

DETAILS: All outbound messages are being quarantined due to elevated spam scoring. Proofpoint is working to correct mail flow delays.

UPDATE: Next update in 30 minutes

Status remaining
Major Outage
2020-08-11T17:40:17.119968+00:00

Getting calls about outbound delivery issues ... messages are getting blocked by Proofpoint.

We are in the process of getting Proofpoint to resolve this issue. Will send update as soon as we have one.

Status changed to
Major Outage
2020-08-11T17:23:04.629719+00:00
Back to Current Status