https://www.fastly.com/blog/summary-of-june-8-outage

  * EN

  * 

  * (844) 4FASTLY
  * Support
  * Log in

Fastly logoCall us Try Fastly free Toggle Navigation

  * Why Fastly
  * Products
  * Solutions
  * Resources
  * Pricing

  * (844) 4FASTLY
  * Support
  * Log in

  * EN

Talk to an expert Try Fastly free

Summary of June 8 outage

Published June 8, 2021

  * Company news
  * Network

[speaker-ni]
Nick Rockwell
Senior Vice President of Engineering and Infrastructure

We experienced a global outage due to an undiscovered software bug
that surfaced on June 8 when it was triggered by a valid customer
configuration change. We detected the disruption within one minute,
then identified and isolated the cause, and disabled the
configuration. Within 49 minutes, 95% of our network was operating as
normal.

This outage was broad and severe, and we're truly sorry for the
impact to our customers and everyone who relies on them.

What happened?

On May 12, we began a software deployment that introduced a bug that
could be triggered by a specific customer configuration under
specific circumstances.

Early June 8, a customer pushed a valid configuration change that
included the specific circumstances that triggered the bug, which
caused 85% of our network to return errors.

Here's a timeline of the day's activity (all times are in UTC): 

09:47 Initial onset of global disruption
09:48 Global disruption identified by Fastly monitoring
09:58 Status post is published
10:27 Fastly Engineering identified the customer configuration
10:36 Impacted services began to recover
11:00 Majority of services recovered
12:35 Incident mitigated
12:44 Status post resolved
17:25 Bug fix deployment began

Once the immediate effects were mitigated, we turned our attention to
fixing the bug and communicating with our customers. We created a
permanent fix for the bug and began deploying it at 17:25.

Where do we go from here? 

In the short term:

  * We're deploying the bug fix across our network as quickly and
    safely as possible. 

  * We are conducting a complete post mortem of the processes and
    practices we followed during this incident. 

  * We'll figure out why we didn't detect the bug during our software
    quality assurance and testing processes.

  * We'll evaluate ways to improve our remediation time.

We have been -- and will continue to -- innovate and invest in
fundamental changes to the safety of our underlying platforms.
Broadly, this means fully leveraging the isolation capabilities of
WebAssembly and Compute@Edge to build greater resiliency from the
ground up. We'll continue to update our community as we make progress
toward this goal.

Conclusion

Even though there were specific conditions that triggered this
outage, we should have anticipated it. We provide mission critical
services, and we treat any action that can cause service issues with
the utmost sensitivity and priority. We apologize to our customers
and those who rely on them for the outage and sincerely thank the
community for its support. Customers should always feel free to email
support@fastly.com for more information. 

Get a Fastly demo
Share this post
facebookTwitterLinked In
[speaker-ni]
Nick Rockwell
Senior Vice President of Engineering and Infrastructure

Nick Rockwell is the Senior Vice President of Engineering and
Infrastructure at Fastly, with 20 years of experience in product
development and information security at top media publishing
companies. Previously, Nick served as Chief Technology Officer at The
New York Times, where he led the company's digital transformation
through a massive re-architecture of its website and other product
offerings.

Subscribe to our newsletter

Get the latest news and industry insights in your inbox.

[                                                            ]
Subscribe
[ ]We send emails from time to time about upcoming events, new
products, & the latest industry trends, but only if you want them.
Check the box to consent to receive these messages.

Get a Fastly demo

By submitting your request, you consent to your information being
transmitted to Fastly in the United States for processing consistent
with our Privacy Policy.

Ready to get started?

Get in touch or create an account.

Try Fastly free Talk to an expert

  * Products
  * For Developers
  * For Businesses
  * Company
  * Support

Fastly logo

  * Acceptable use
  * Terms of service
  * Privacy policy
  * Website data collection
  * CCPA compliance statement
  * Code of conduct

(c)2021 Fastly, Inc. All Rights Reserved

FacebookLinkedInTwitter