NexusNation
NexusNationSYSTEMSDEV
← All incidents

Production Incident Atlas

Cloudflare’s July 2019 outage: a WAF rule exhausted CPU worldwide

A globally deployed WAF rule contained a regular expression with excessive backtracking, exhausting CPU on Cloudflare’s HTTP/HTTPS serving path.

Cloudflare WAF and global networkSoftware & infrastructureSoftware defectMajor severityAvailability

What happened

Incident timeline

6 documented events

  1. A minor change to XSS detection rules was deployed through the automatic WAF rules process.

  2. The first PagerDuty alert reported a WAF synthetic-test failure; other end-to-end failures, a global traffic-drop alert, 502 errors, and CPU-exhaustion reports followed.

  3. Responders identified the WAF as the source of the problem and dismissed an attack as the cause.

  4. An engineer executed the global WAF termination mechanism to disable the affected component worldwide.

  5. Cloudflare reported traffic levels and CPU back to expected worldwide levels; other protection mechanisms remained active.

  6. After testing the cause and rollback on a subset of traffic, Cloudflare re-enabled the WAF globally.

Detection

A WAF synthetic test paged responders three minutes after deployment. Further end-to-end test failures, a global traffic-drop alert, widespread 502 errors, and CPU-exhaustion reports from points of presence followed. Responders identified the WAF as the cause at 14:00 UTC.

Recovery

A global WAF termination was executed at 14:07 UTC. Cloudflare reported traffic and CPU back to expected levels worldwide by 14:09 UTC. After testing the rollback with a subset of traffic, the WAF was re-enabled globally at 14:52 UTC.

Lessons afterward

Cloudflare’s postmortem explains that simulate mode still executes rules, while its WAF test suite did not test for runaway CPU use. It also records that responders could not use the normal internal control panel while Access was down and had to use a bypass mechanism that was not frequently practiced.

Editorial analysis

Analysis

The immediate trigger was a regular expression with excessive backtracking, but the postmortem describes a chain of conditions around it. The WAF rule was deployed globally because that rollout path was designed to respond quickly to emerging threats. Its “simulate” mode still evaluated the expression against live requests, and the test suite did not measure runaway CPU use.

Containment was fast once the team identified the WAF and invoked the global termination mechanism: Cloudflare reported traffic and CPU returning to expected levels two minutes later. Restoring the WAF itself took longer because the team tested the cause and rollback on a subset of traffic before enabling it globally.

The account also documents a response dependency: Cloudflare’s own Access service was unavailable, blocking the usual internal control panel and forcing responders to use a less-practiced bypass. This is a reminder to review both software safeguards and independent access paths needed during an outage.

Last reviewed 10/9/2026 by Nexus Lead Architect.

Change notes

  • 10/9/2026: Initial publication from https://blog.cloudflare.com/details-of-the-cloudflare-outage-on-july-2-2019/

Corrections or additional primary evidence? Contact the editorial team.

Cloudflare’s 2019 Global Outage: WAF Rule Timeline and Lessons — NexusNation