Microsoft says a software defect introduced in July, a later increase in network traffic and a temporary loss of server capacity combined to cause the global Microsoft 365 outage on November 12–13. BleepingComputer’s November 16 report on Microsoft’s preliminary post-incident review for incidents EX1550543 and MO1550821 says the problems disrupted Exchange Online mailbox access and delayed email delivery.

The report says the defect was a lock-contention problem inside an internal Windows process running on Microsoft 365 networking gateway servers. The code change had been deployed worldwide over several weeks and operated for more than three months without visible impact. Microsoft said its load testing missed the issue because its labs focused on weaker hardware with fewer cores, while the contention became worse on newer machines with more cores.

About 15 hours before the outage, Teams traffic failed over from an alternative gateway to the primary gateway shared with Exchange Online. At the same time, nearly 14% of that gateway’s server capacity was unavailable because of a separate issue from the previous day. Microsoft said the reduced capacity was not enough to cause an outage on its own, but the combination of increased traffic and reduced capacity exposed the latent defect. Automated traffic redirection then pushed requests to remaining endpoints, which encountered the same problem. In less than two hours, more than 80% of the gateway servers were unresponsive, and the usual safeguards between capacity partitions failed to contain the cascade.

READ
Microsoft 365 Outage Disrupts Outlook and Teams Worldwide as Recovery Begins

Recovery took longer because servers became unresponsive again after being restarted and receiving queued traffic. Microsoft’s normal recovery and rollback processes were not designed for such a broad failure, and the underlying code change was several months old. Engineers eventually deployed an update that returned the connection-handling code path to a previously known-good state, applying it to groups of servers before restarting them. The preliminary report says a final post-incident review will follow within five business days of full resolution.


Buy ExpressVPN with PayPal or Credit Card
Advertisement