In a more detailed post published Tuesday October 5th, Santosh Janardhan, Facebook’s vice president of infrastructure, said in a blog post that… Facebook, Instagram and WhatsApp going dark was “caused not by malicious activity, but an error of our own making.”
Santosh Janardhan said there was a “bug” in a tool meant to prevent mistakes like what triggered the outage from happening. Facebook encountered multiple problems, including getting access to its data centers and domain name system servers, which had become unreachable. Referred to as the phone book of the internet, DNS translates domain names like Facebook.com to numeric Internet Protocol addresses. “The total loss of DNS broke many of the internal tools we’d normally use to investigate and resolve outages like this,” Janardhan said.
Janardhan, then went on to state… “Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication,” Facebook’s vice president of engineering and infrastructure, Santosh Janardhan, said in a blog post. “This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt.”
The rare outage, which also impacted other apps owned by Facebook such as Instagram, WhatsApp, and Facebook Messenger, showcased how dependent people and businesses are on social media. Every failure like this is an opportunity to learn and get better. Janardhan then said, “After every issue, small and large, we do an extensive review process to understand how we can make our systems more resilient. That process is already underway.”