What is Wrong with Facebook today

What Is Wrong With Facebook Today - Early today Facebook was down or inaccessible for many of you for approximately 2.5 hrs. This is the worst interruption we have actually had in over 4 years, as well as we wished to firstly excuse it. We likewise wanted to give a lot more technical detail on what took place and also share one huge lesson learned.

What's Wrong With Facebook

What Is Wrong With Facebook Today


The essential flaw that caused this outage to be so extreme was a regrettable handling of a mistake problem. An automated system for verifying arrangement worths ended up creating a lot more damage than it dealt with.

The intent of the automated system is to look for arrangement worths that are void in the cache as well as replace them with updated worths from the persistent shop. This works well for a short-term trouble with the cache, but it doesn't work when the relentless shop is void.

Today we made a modification to the persistent duplicate of an arrangement worth that was interpreted as invalid. This meant that every single client saw the void worth as well as tried to repair it. Because the repair entails making an inquiry to a cluster of databases, that collection was promptly overwhelmed by hundreds of countless questions a second.

To make issues worse, every time a client got an error attempting to quiz one of the databases it analyzed it as a void value, and also removed the matching cache key. This meant that even after the original issue had been dealt with, the stream of inquiries proceeded. As long as the databases failed to service several of the requests, they were triggering even more demands to themselves. We had entered a comments loophole that really did not allow the databases to recover.

The way to stop the responses cycle was fairly excruciating - we had to stop all website traffic to this database cluster, which meant switching off the website. Once the data sources had actually recouped and also the origin had actually been taken care of, we gradually enabled even more people back onto the website.

This obtained the website back up and also running today, as well as in the meantime we have actually turned off the system that attempts to fix configuration values. We're exploring new designs for this configuration system following style patterns of various other systems at Facebook that deal even more with dignity with feedback loopholes and transient spikes.

We apologize again for the site failure, and we want you to know that we take the efficiency and also reliability of Facebook extremely seriously.