What is Wrong with Facebook

What Is Wrong With Facebook - Early today Facebook was down or unreachable for a number of you for approximately 2.5 hours. This is the worst outage we have actually had in over 4 years, as well as we wanted to first off apologize for it. We also wanted to supply a lot more technological detail on what occurred as well as share one huge lesson discovered.

What's Wrong With Facebook

What Is Wrong With Facebook


The key problem that triggered this interruption to be so serious was an unfavorable handling of a mistake condition. An automated system for verifying arrangement worths ended up creating much more damage than it dealt with.

The intent of the computerized system is to check for setup values that are invalid in the cache as well as change them with updated values from the persistent store. This functions well for a transient issue with the cache, yet it does not work when the consistent shop is invalid.

Today we made an adjustment to the persistent copy of a setup worth that was interpreted as void. This implied that every single customer saw the void value and attempted to fix it. Because the repair includes making a question to a cluster of databases, that collection was quickly bewildered by numerous thousands of questions a 2nd.

To make issues worse, each time a customer obtained an error attempting to inquire among the data sources it interpreted it as an invalid value, as well as removed the equivalent cache key. This meant that even after the initial trouble had actually been taken care of, the stream of questions continued. As long as the databases failed to service a few of the requests, they were triggering much more requests to themselves. We had gone into a feedback loop that really did not enable the data sources to recover.

The way to stop the responses cycle was rather agonizing - we had to stop all website traffic to this database cluster, which indicated shutting off the website. As soon as the databases had actually recovered and the source had been fixed, we slowly allowed more people back onto the site.

This got the website back up and also running today, as well as in the meantime we've switched off the system that tries to fix setup values. We're checking out brand-new designs for this arrangement system complying with style patterns of various other systems at Facebook that deal more with dignity with feedback loopholes as well as short-term spikes.

We ask forgiveness again for the website blackout, as well as we want you to know that we take the performance and also reliability of Facebook extremely seriously.