Fast.ly broke the Internet for an hour this morning

[DudeAsInCool]

Enlarge / The United Kingdom government's official website was one of those affected by this morning's outage. The cryptic "Guru Mediation" message shown is an untrapped, unskinned error returned from the Varnish cache server powering the Fastly CDN. (credit: Leon Neal via Getty Images)

For roughly an hour this morning—6 am to 7 am EDT, give or take a few minutes—enormous swathes of the Internet were down or interestingly broken. Sites taken down included CNN, The Guardian, The New York Times, PayPal, and Spotify, among many more—including The Verge, which resorted to reporting via Google Docs during the duration of the outage.

Vast chunks of the internet are offline, including The Verge. Until we’re back, we’re reporting to you live out of Google Docs. Here’s what we know so far about the outage: https://t.co/4b1p2qhYif

— The Verge (@verge) June 8, 2021

The underlying problem was an outage at Fastly, one of the world's largest Content Delivery Network providers—the entire service went down due to a misconfiguration which it had deployed to all of its Points Of Presence (POPs) globally. As a result, sites using Fastly for content delivery came up with various errors dependent on the local site configuration. Some sites delivered relatively uninformative, plain HTTP 503 (Service Unavailable) pages, while others returned errors such as "Fastly error: unknown domain."

The "unknown domain" error gives us some tantalizing hints to the nature of the problem, which is more than Fastly's own status updates have so far. This tells us that Fastly's network was up and its Varnish cache servers were answering requests, but its cache configuration—the Varnish Control Language files which point the cache server to the back-end servers supplying the original content—was almost certainly either missing or garbled.

Read 3 remaining paragraphs | Comments

View the full article