Cloudflare CEO explains exactly what caused global outage
On Tuesday, a significant outage at Cloudflare disrupted access to a vast array of popular websites and online services, including X (formerly Twitter), ChatGPT, Spotify, YouTube, and Uber. This incident marked the worst outage for Cloudflare since 2019, as noted by co-founder and CEO Matthew Prince in a blog post addressing the situation. Prince expressed his apologies for the inconvenience caused, stating, “On behalf of the entire team at Cloudflare, I would like to apologize for the pain we caused the Internet today.” The incident has raised questions about the resilience of major internet infrastructure and the frequency of such disruptions, especially following recent outages from other cloud service providers like Google Cloud and AWS.
The root cause of the outage was traced back to a malfunction in Cloudflare’s Bot Management system, which is designed to protect websites from various types of malicious bot attacks, including DDoS attacks and content scraping. This system relies on an AI model that assesses traffic requests by generating a “feature file,” which is updated every five minutes to adapt to evolving bot behaviors. However, an update to the underlying query responsible for generating this feature file resulted in duplicated information, causing the file to exceed its typical size. This anomaly triggered errors in the Bot Management system, leading to widespread access issues for users attempting to reach websites protected by Cloudflare. Initially, the company suspected that the outage might be the result of a cyber attack, as their status page also experienced downtime. However, Prince clarified that the outage was not linked to any malicious activity, emphasizing that the symptoms observed were coincidental.
Cloudflare’s services began to recover within three hours, with full restoration achieved after approximately five hours. In light of this incident, Prince indicated that the company is taking steps to fortify its systems against future outages, including implementing measures to prevent error reports from overwhelming their infrastructure. This outage serves as a stark reminder of the vulnerabilities inherent in the internet’s foundational services and the importance of robust cybersecurity measures in maintaining seamless online experiences for users around the globe.
https://www.youtube.com/watch?v=I7Ov-Q-0h5c
A
Cloudflare outage
took out
a large swathe of the internet on Tuesday
, with users unable to access numerous sites and services such as
X
,
ChatGPT
,
Spotify
,
YouTube
, and
Uber
. The cybersecurity company has now published a
blog post
detailing exactly what happened.
SEE ALSO:
Why does the internet keep crashing so often? First Google Cloud, then AWS, now Cloudflare.
Cloudflare co-founder and CEO Matthew Prince apologised in the post late Tuesday, stating that this outage was the worst the company has experienced since 2019.
“[I]n the last 6+ years we’ve not had another outage that has caused the majority of core traffic to stop flowing through our network,” said Prince. “On behalf of the entire team at Cloudflare, I would like to apologize for the pain we caused the Internet today.”
Prince explained that the Cloudflare outage had been caused by an issue with the system it uses to protect websites from DDoS attacks.
Cloudflare’s outage, explained
This Tweet is currently unavailable. It might be loading or has been removed.
Cloudflare’s Bot Management system
is a service which protects websites against malicious bot attacks. These include
DDoS attacks
that flood websites with excessive traffic,
content scraping
attacks which gather data from websites without authorisation, and autonomous credential stuffing attacks which try to gain access to websites by using leaked login details from other sites.
This Bot Management system includes an AI model which scores traffic requests. Whenever there’s an attempt to access a website protected by Cloudflare’s Bot Management, the AI generates a score to determine if it’s likely to have been from a bot. In order to do so, the AI considers various features of the request, which are held in a “feature file.”
The feature file is where the issue occurred. This file refreshes every five minutes to keep up to date with evolving bot behaviours, and is used across Cloudflare’s entire cybersecurity network. However, the company implemented a change to the underlying query that generated the file, which caused it to duplicate information a large number of times. This made the feature file larger than typical, triggering an error in the Bot Management system.
As a result, attempting to access websites which use Cloudflare’s Bot Management system resulted in an error code. Cloudflare states that its network began experiencing significant failures about 15 minutes after the feature file generation update was implemented.
Cloudflare initially suspected the outage was a malicious attack, particularly as its status page went down despite being independent from the company’s infrastructure. However, Prince stated that this turned out to be a coincidence.
“The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind,” Prince stressed. “After we initially wrongly suspected the symptoms we were seeing were caused by a hyper-scale DDoS attack, we correctly identified the core issue and were able to stop the propagation of the larger-than-expected feature file and replace it with an earlier version of the file.”
When previously reached by Mashable prior to the blog post, a Cloudflare spokesperson also emphasised that
“there [was] no evidence that [the outage] was the result of an attack or caused by malicious activity.”
Cloudflare’s services were largely restored within three hours, and fully restored after approximately five hours. Prince stated that the company is already planning measures to prevent similar outages in the future, including stopping error reports from being able to overwhelm its systems.
