Cloudflare CEO explains exactly what caused global outage
On Tuesday, a significant outage at Cloudflare disrupted access to a wide array of popular websites and services, including X (formerly Twitter), ChatGPT, Spotify, YouTube, and Uber, leaving users frustrated across the globe. Matthew Prince, co-founder and CEO of Cloudflare, acknowledged the severity of the incident in a blog post, labeling it the worst outage the company has faced since 2019. He expressed his apologies, stating that it was an unprecedented disruption, with the majority of core internet traffic coming to a halt. The outage was traced back to a malfunction in Cloudflare’s Bot Management system, which is designed to protect websites from various forms of cyber threats, including Distributed Denial of Service (DDoS) attacks.
The root cause of the outage was linked to an issue with the “feature file,” a critical component that the Bot Management system uses to assess incoming traffic requests. This file, which refreshes every five minutes to adapt to evolving bot behaviors, encountered a problem after a change was made to the query generating it. This alteration inadvertently caused the file to duplicate data excessively, resulting in an error that disrupted the Bot Management system’s functionality. Cloudflare’s systems began to fail approximately 15 minutes after the update, leading to widespread access issues. Initially, the company suspected that the outage might have been caused by a malicious DDoS attack, especially since its status page, which is supposed to be independent of its infrastructure, also went down. However, Prince later clarified that there was no evidence of any cyber attack involved, and the issue was purely technical.
Cloudflare managed to restore most of its services within three hours, with full functionality returning after approximately five hours. In light of this incident, Prince announced that the company is taking steps to prevent similar occurrences in the future. These measures include implementing safeguards to prevent error reports from overwhelming its systems, ensuring that such a disruption does not happen again. This incident serves as a stark reminder of the fragility of internet infrastructure and the critical role companies like Cloudflare play in maintaining online accessibility and security. As more services rely on cloud-based solutions, the implications of such outages can ripple across the digital landscape, affecting countless users and businesses worldwide.
https://www.youtube.com/watch?v=I7Ov-Q-0h5c
A
Cloudflare outage
took out
a large swathe of the internet on Tuesday
, with users unable to access numerous sites and services such as
X
,
ChatGPT
,
Spotify
,
YouTube
, and
Uber
. The cybersecurity company has now published a
blog post
detailing exactly what happened.
SEE ALSO:
Why does the internet keep crashing so often? First Google Cloud, then AWS, now Cloudflare.
Cloudflare co-founder and CEO Matthew Prince apologised in the post late Tuesday, stating that this outage was the worst the company has experienced since 2019.
“[I]n the last 6+ years we’ve not had another outage that has caused the majority of core traffic to stop flowing through our network,” said Prince. “On behalf of the entire team at Cloudflare, I would like to apologize for the pain we caused the Internet today.”
Prince explained that the Cloudflare outage had been caused by an issue with the system it uses to protect websites from DDoS attacks.
Cloudflare’s outage, explained
This Tweet is currently unavailable. It might be loading or has been removed.
Cloudflare’s Bot Management system
is a service which protects websites against malicious bot attacks. These include
DDoS attacks
that flood websites with excessive traffic,
content scraping
attacks which gather data from websites without authorisation, and autonomous credential stuffing attacks which try to gain access to websites by using leaked login details from other sites.
This Bot Management system includes an AI model which scores traffic requests. Whenever there’s an attempt to access a website protected by Cloudflare’s Bot Management, the AI generates a score to determine if it’s likely to have been from a bot. In order to do so, the AI considers various features of the request, which are held in a “feature file.”
The feature file is where the issue occurred. This file refreshes every five minutes to keep up to date with evolving bot behaviours, and is used across Cloudflare’s entire cybersecurity network. However, the company implemented a change to the underlying query that generated the file, which caused it to duplicate information a large number of times. This made the feature file larger than typical, triggering an error in the Bot Management system.
As a result, attempting to access websites which use Cloudflare’s Bot Management system resulted in an error code. Cloudflare states that its network began experiencing significant failures about 15 minutes after the feature file generation update was implemented.
Cloudflare initially suspected the outage was a malicious attack, particularly as its status page went down despite being independent from the company’s infrastructure. However, Prince stated that this turned out to be a coincidence.
“The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind,” Prince stressed. “After we initially wrongly suspected the symptoms we were seeing were caused by a hyper-scale DDoS attack, we correctly identified the core issue and were able to stop the propagation of the larger-than-expected feature file and replace it with an earlier version of the file.”
When previously reached by Mashable prior to the blog post, a Cloudflare spokesperson also emphasised that
“there [was] no evidence that [the outage] was the result of an attack or caused by malicious activity.”
Cloudflare’s services were largely restored within three hours, and fully restored after approximately five hours. Prince stated that the company is already planning measures to prevent similar outages in the future, including stopping error reports from being able to overwhelm its systems.
