Something Wrong with Facebook

Something Wrong With Facebook - Early today Facebook was down or unreachable for a number of you for approximately 2.5 hrs. This is the worst outage we have actually had in over four years, and we wanted to first of all excuse it. We additionally wished to provide a lot more technological information on what occurred as well as share one huge lesson learned.

What's Wrong With Facebook

Something Wrong With Facebook


The key flaw that created this outage to be so serious was an unfavorable handling of an error condition. A computerized system for confirming arrangement values ended up triggering much more damages than it repaired.

The intent of the automatic system is to check for arrangement values that are void in the cache and also change them with updated worths from the persistent shop. This works well for a short-term issue with the cache, yet it doesn't work when the persistent store is invalid.

Today we made an adjustment to the relentless copy of an arrangement worth that was taken void. This suggested that every single client saw the void worth as well as attempted to repair it. Because the solution entails making a question to a collection of data sources, that cluster was rapidly bewildered by numerous countless questions a 2nd.

To make matters worse, every time a client got a mistake trying to quiz among the databases it translated it as an invalid worth, and also deleted the corresponding cache key. This suggested that also after the original trouble had actually been taken care of, the stream of queries continued. As long as the data sources failed to service several of the demands, they were causing a lot more demands to themselves. We had actually entered a comments loop that really did not allow the data sources to recover.

The way to quit the responses cycle was rather agonizing - we needed to stop all web traffic to this data source cluster, which suggested switching off the site. When the data sources had actually recouped as well as the source had actually been dealt with, we gradually permitted more individuals back onto the website.

This obtained the site back up and running today, as well as in the meantime we have actually turned off the system that tries to correct setup worths. We're exploring new styles for this arrangement system complying with design patterns of various other systems at Facebook that deal even more with dignity with responses loops and transient spikes.

We say sorry once more for the site blackout, and we want you to recognize that we take the performance and also dependability of Facebook very seriously.