Skip to content
Technology

The Microsoft 365 outage was caused weeks before it broke

The Deploy Desk 7 min read

On Thursday, November 12, Outlook went dark for organizations in more than 70 countries. According to Microsoft, the root of the problem was not that day’s change. It was an earlier network capacity upgrade that sat quietly in production until a later change woke it up.

That detail is why this outage is worth more than a shrug. Most of us will never run anything at Microsoft’s scale, but every team that ships changes has the same blind spot: we only suspect the last deploy. Here is what happened, and five lessons you can apply to your own pipeline this week.

What happened

Exchange Online, the service behind Outlook, failed on November 12 and took most of two days to fully recover. Microsoft has not published its final post-incident report yet, so treat the cause below as its preliminary explanation.

When (UTC)What happened
Nov 12, ~16:17Exchange Online incident begins (EX1550543). Users can’t open mailboxes on web, desktop or mobile.
Nov 12, within an hourOutage reports arrive from 74 countries and territories. Downdetector peaks at roughly 138,000 Outlook reports.
Nov 12, eveningMicrosoft reroutes traffic and restarts infrastructure, with only limited recovery.
Nov 13, morningRestored servers fail again once traffic returns to them.
Nov 13, ~07:30Microsoft reports positive results after patching an “upstream authentication/routing configuration” component. About three hours later, service starts returning region by region, Asia Pacific first.
Nov 14Microsoft says the underlying issue is addressed and remaining customers are restored.

The blast radius went well beyond email. Teams and Microsoft 365 Copilot features that rely on mailbox data broke, Intune admins hit errors, and third-party integrations built on Microsoft 365 failed with them. On-premises Exchange Server was not affected.

Microsoft’s explanation so far has three parts:

  • Cause: a problem introduced during an earlier network capacity upgrade, which surfaced after a more recent network change.
  • Symptom: elevated load across several regions and servers becoming unexpectedly unavailable. Microsoft said the load was not caused by more user traffic.
  • Not the cause: a cyberattack. Microsoft said there was no sign of malicious activity.

Lesson 1: Your riskiest change might be an old one

When something breaks, the first question is always “what did we just deploy?” That instinct is right most of the time, which is exactly why latent bugs are so expensive. If the bad change shipped two weeks ago and passed every check, your rollback of today’s change may fix nothing.

A change can sit harmlessly in production for a long time. It only fails when a second change, a traffic pattern or a failover finally exercises the broken path. Capacity upgrades and network changes are classic examples, because their failure modes often only show up under specific load.

What to do about it:

  • Keep one change log for everything. App deploys, infra changes, config flags, DNS, certificates and vendor updates, all with timestamps. During an incident you want to scan the last 30 days, not just the last hour.
  • Record what each change touched. A one-line “blast radius” note (which services, which regions) makes correlation fast.
  • Treat “we already rolled back” as a hypothesis. If symptoms continue after rollback, widen the window instead of digging deeper into the latest change.

Lesson 2: Recovery is a deploy too

The most painful part of this incident was not the first failure. Microsoft restarted servers, and some went unresponsive again as soon as traffic came back to them. That is a pattern almost every on-call engineer has seen, often at 3 a.m.

When a service is down, demand piles up behind it: clients retrying, queues growing, users hitting refresh. The first healthy server to come back receives all of that at once and falls over. Then the next one does too. Restarting faster doesn’t help, because each server dies the same way.

Treat recovery like a rollout:

  1. Bring capacity back before you open the doors. Restore enough servers to handle the backlog, not just normal traffic.
  2. Ramp traffic gradually. Send 5%, then 25%, then 50% through your load balancer or feature flags, and watch error rates at each step.
  3. Restore one region or tenant group at a time. Microsoft’s recovery went region by region, starting with Asia Pacific. Smaller steps mean a failed step hurts fewer users.
  4. Drain the backlog on purpose. Rate-limit queue consumers so the backlog clears at a speed your system can handle.

If you have never practiced this, write it down now as a runbook. Nobody designs a good ramp-up plan in the middle of an outage.

Lesson 3: Make your retries polite

Retries are where your own code joins the stampede. A client that retries immediately, forever, on every error turns a partial outage into a full one. Multiply that by thousands of clients and you get the “restored server falls over again” loop from Lesson 2.

Three rules fix most of it:

  • Back off exponentially. Wait longer after each failed attempt.
  • Add jitter. Randomize the wait so clients don’t all retry at the same moment.
  • Give up eventually, and only retry what is retryable. A 400 will never succeed on the second try. A 429, a 5xx or a network error might.

Here is a small TypeScript helper that does all three, using the “full jitter” approach:

async function withRetry<T>(
  fn: () => Promise<T>,
  { retries = 5, baseMs = 200, maxMs = 10_000 } = {}
): Promise<T> {
  for (let attempt = 0; ; attempt++) {
    try {
      return await fn();
    } catch (err) {
      if (attempt >= retries || !isRetryable(err)) throw err;
      const cap = Math.min(maxMs, baseMs * 2 ** attempt);
      const delay = Math.random() * cap; // full jitter
      await new Promise((resolve) => setTimeout(resolve, delay));
    }
  }
}

function isRetryable(err: unknown): boolean {
  const status = (err as { status?: number })?.status;
  // No status usually means a network error: worth retrying.
  return status === undefined || status === 429 || status >= 500;
}

If a server sends a Retry-After header, respect it instead of your own delay. And if you call a dependency from many places, put a circuit breaker in front of it so that after repeated failures you stop calling it for a while.

Lesson 4: Health checks should protect you, not finish you off

During the recovery, Microsoft said it was looking at changing its health checks. That is a telling detail. Health checks decide which servers get traffic, so a badly designed one can make an outage worse.

Two common traps:

  • The check is too shallow. It returns 200 while the app can’t reach its database, so the load balancer keeps sending users to a broken server.
  • The check is too deep. Every server checks a shared dependency. When that dependency slows down, every server marks itself unhealthy at once, and the load balancer has nowhere left to send traffic.

A safer setup splits the job in two. A liveness check answers “is this process alive?” and restarts it if not. A readiness check answers “can this instance take traffic right now?” and removes it from rotation if not. Keep shared dependencies out of liveness checks.

Then add load shedding. When a server is overloaded, it should quickly reject extra requests with a 503 and a Retry-After header, instead of slowly failing all of them. A fast “no” keeps the server alive for the requests it can still serve.

Lesson 5: Your vendor’s outage is your outage

Many of the companies hit last week had nothing wrong with their own systems. Maersk and Euronext reported communication problems, one company postponed its earnings call, and integrations at AI providers like OpenAI and Anthropic broke because they depended on Microsoft 365. To their users, it didn’t matter whose server failed.

You can’t prevent a vendor outage, but you can decide in advance how much it hurts:

  • Map your critical dependencies. List every external service whose failure would stop sign-in, payments, alerts or customer communication.
  • Have a backup channel for incidents. If your on-call alerts, status updates and team chat all run through one provider, an outage there leaves you blind exactly when you need to talk.
  • Degrade gracefully. If the mailbox or calendar integration is down, show cached data or a clear message instead of crashing the whole page.
  • Put timeouts on every external call. A dependency that hangs is worse than one that fails, because it ties up your threads and workers.
  • Subscribe to your vendors’ status pages. Route them into the same channel as your own alerts, so you’re not debugging your code while the problem is upstream.

TL;DR: a checklist for this week

You don’t need Microsoft’s scale to get value from Microsoft’s bad week. Pick two of these and do them before Friday:

  • Put every infra, config and app change into one timestamped change log
  • Write a recovery runbook that ramps traffic back in steps (5% → 25% → 50% → 100%)
  • Audit your retry logic: exponential backoff, jitter, a retry limit, and respect for Retry-After
  • Split health checks into liveness and readiness, and keep shared dependencies out of liveness
  • Add load shedding: return a fast 503 when overloaded instead of failing slowly
  • List your critical vendors and set up a backup channel for incident communication

Microsoft has said a detailed post-incident report will follow. When it’s out, I’ll update this post with what it reveals.

Sources