Skip to content

Cloud & Infrastructure · Reliability

The AWS CloudFront Outage That Took Down Canvas, Blackboard, and Hugging Face at Once

A 3-hour-33-minute CloudFront VPC Origins failure on July 16, 2026 knocked out at least ten unrelated services worldwide. The root cause was a single-ingress design decision, and it's a pattern worth checking your own architecture for.

Shashikant Gupta

Shashikant Gupta

5 min read

The AWS CloudFront Outage That Took Down Canvas, Blackboard, and Hugging Face at Once

Sponsored

Share

For 3 hours and 33 minutes on July 16, 2026, a single misconfiguration in one AWS availability zone took down university learning platforms, an AI infrastructure company, a national lottery, and a networking hardware vendor’s cloud services, all at once, all unrelated to each other except for one shared dependency. That’s the CloudFront VPC Origins outage, and the interesting part isn’t the outage itself. It’s the architectural choice that turned a contained fault into a global one.

What broke, and where

AWS’s root cause analysis traced the failure to an internal constraint on the fleet that manages connections to CloudFront’s private VPC Origins feature, specifically in the euc1-az2 zone of EU-CENTRAL-1, AWS’s Frankfurt region. When that constraint was hit, the system responsible for distributing updated routing configuration to network processors failed to load the configuration correctly. From 07:45 to 11:18 UTC, CloudFront returned 5xx errors to every customer using VPC Origins, not just customers with origins in Frankfurt.

Diagram showing the 3-hour-33-minute CloudFront VPC Origins outage window from 07:45 to 11:18 UTC on July 16, 2026, alongside a list of reported affected services including Canvas, Blackboard, Hugging Face, the UK National Lottery, and Ubiquiti Cloud

The list of affected services reads like a random sample of the internet: Canvas and Blackboard, the learning management systems universities across North America rely on for coursework and grading, went offline during exam season for some institutions. Hugging Face was disrupted, in what turned out to be an unrelated coincidence days before its own security incident made headlines. The UK National Lottery’s infrastructure went down. Ubiquiti’s cloud services for networking hardware were affected. None of these companies share a codebase, a team, or a business model. They share a CDN feature.

The design decision that turned a zone issue into a global one

Here’s the part worth sitting with. VPC Origins exists specifically to improve security: it lets CloudFront reach a private origin inside a customer’s VPC without that origin needing a public IP address or exposure to the open internet. That’s a real improvement over the alternative, and it’s why the feature exists. But it comes with a tradeoff that’s easy to miss until an incident like this makes it visible: a private connection with no alternate network path also has no alternate network path when something goes wrong with it.

Normal CloudFront routing to a public origin has path diversity baked in, multiple ways to reach the same endpoint, which is what usually contains a regional fault to that region. VPC Origins deliberately narrows that to a single private path as part of its security model. When the fleet managing that single path failed in one zone, there was no fallback route for it to fail over to, so the failure propagated to every customer depending on that path, everywhere, instead of staying contained to customers with origins physically located in the affected zone.

This is not a criticism of the security design. A tighter, single network path is often exactly the right tradeoff for a private origin that shouldn’t be reachable any other way. The mistake, if there is one, is treating that tradeoff as free. It isn’t. You’re trading path diversity, and therefore some blast-radius containment, for a tighter security boundary, and an outage like this is what happens when the assumption “this path won’t fail” turns out to be wrong.

The pattern to check in your own architecture

This same tradeoff shows up outside of CloudFront specifically, anywhere an architecture chooses a single, tightly-scoped network path over a more open but more redundant one. Private links to a database, a service mesh sidecar that’s the only route to a backend, a VPN tunnel that’s the sole path between two environments, all of these make the same bet VPC Origins makes: better security in exchange for a single point of failure at the network layer. That bet is often correct. The question worth asking is whether anyone explicitly made that decision and planned for the failure mode, or whether it was inherited by default because the security-focused option was also the modern-looking one.

We’ve written before about what a similar single-provider dependency looked like when Cloudflare had its own outage, and the underlying lesson repeats here with a different vendor and a different mechanism: the failure that takes down your service is rarely the origin server itself. It’s almost always the network path you didn’t think to test, because it was chosen for reasons that had nothing to do with availability.

What to actually do about it

If you use CloudFront VPC Origins, or an equivalent single-path private connection anywhere in your stack, the useful exercise isn’t to abandon the security benefit. It’s to explicitly decide, for each workload, whether that single path is an acceptable single point of failure or not. For most workloads it is, a few hours of degraded service is a real cost but not a business-ending one. For anything where it isn’t, a documented fallback, even a degraded one like serving cached content or a maintenance page, beats a hard failure with no plan. Test that fallback the same way you’d test origin failover, because right now, most teams have tested the second and not the first. A reliability review that specifically maps single-path dependencies rather than just single-origin ones is the kind of exercise that finds this before an incident does.

Frequently asked questions

What actually broke in the July 16, 2026 CloudFront outage?
AWS's root cause analysis pointed to an internal constraint on the fleet that manages connections to CloudFront's private VPC Origins feature, specifically in the euc1-az2 zone of EU-CENTRAL-1 (Frankfurt). When that constraint was hit, the system responsible for distributing updated routing configuration to network processors failed to load it correctly, and CloudFront began returning 5xx errors to every customer using VPC Origins, regardless of their own region.
Why did a single availability zone issue affect services worldwide?
VPC Origins is built so that traffic to a private origin has no alternate network path, which is a deliberate security tradeoff: it keeps the origin reachable only through CloudFront's private connection rather than exposing it to the public internet. That same design removed the path diversity that normally contains a regional issue, so a fault in one zone's connection-management fleet affected VPC Origins traffic globally rather than staying local to that region.
Was Hugging Face's outage related to its later security breach?
No, they're separate incidents that happened to land in the same week. The CloudFront outage on July 16 was an infrastructure availability failure with no security implications. The Hugging Face breach, where an autonomous AI agent accessed internal data during an OpenAI safety test, was disclosed days later and involved a completely different failure, a vulnerability in Hugging Face's own data-processing pipeline, not a CloudFront issue.
How do I know if my architecture has this same single-ingress risk?
Ask what happens to your service if the specific network path your traffic depends on fails, not what happens if the origin server itself fails. Many architectures test origin failover carefully but never test the failure of the network layer connecting to that origin, especially when that layer was chosen specifically because it has no alternate path, which is exactly the property that made VPC Origins secure and also the property that made this outage global instead of regional.
What should teams using CloudFront VPC Origins do after this?
Confirm whether your application has a fallback path if VPC Origins traffic fails, even a degraded one like a cached error page or a maintenance response, rather than a hard failure. If your origin genuinely cannot tolerate any downtime, a secondary origin reachable through a different mechanism, accepting the reduced security posture that implies, may be the right tradeoff for that specific workload even though it isn't for most.

Sources

Sponsored

Sponsored

Discussion

Join the conversation.

Comments are powered by GitHub Discussions. Sign in with your GitHub account to leave a comment.

Sponsored