California Extreme Heat Takes Key Twitter Data Center Offline | CNN Business

CNN
—

Extreme heat in California left Twitter without one of its main data centers, and a company executive warned in an internal memo obtained by CNN that another outage elsewhere could shut down the service for some of its users.

Twitter (TWTR), like all major social media platforms, relies on data centers, which are basically huge warehouses filled with computers, including servers and storage systems. Temperature control in these centers is essential to ensure computers do not overheat and malfunction. To save on cooling costs, some technology companies have increasingly sought to locate their data centers in colder climates; Google, for example, opened a data center in Finland in 2011, and Meta has had a center in northern Sweden since 2013.

“On September 5, Twitter experienced the loss of its Sacramento Data Center (SMF) region due to extreme weather conditions. The unprecedented event resulted in the complete shutdown of SMF’s physical equipment,” Carrie Fernandez, the company’s vice president of engineering, said in an internal message to Twitter engineers on Friday.

Large tech companies typically have multiple data centers, in part to ensure that their service can stay online if one center fails; this is called redundancy.

Following the outage in Sacramento, Twitter is in a “non-redundant state,” according to Friday’s memo from Fernandez. She explained that Twitter’s data centers in Atlanta and Portland are still operational, but warned, “If we lose one of these remaining data centers, we may not be able to deliver traffic to all of them. Twitter users.”

The memo goes on to ban non-critical product updates from Twitter until the company can fully restore its Sacramento data center services. “All production changes, including deployments and releases on mobile platforms, are blocked except for changes necessary to ensure continuity of service or other urgent operational needs,” Fernandez wrote.

The restrictions highlight the apparent fragility of some of Twitter’s most fundamental systems, an issue that former Twitter security chief turned whistleblower Peiter “Mudge” Zatko raised in a disclosure sent to lawmakers. and to government agencies in July.

In his whistleblower disclosure, first reported by CNN and The Washington Post, Zatko warned that Twitter had “insufficient data center redundancy” which increased the risk of a brief service outage or even the prospect that Twitter disconnect permanently.

“Even a temporary but overlapping outage of a small number of data centers would likely cause service [Twitter] disconnect for weeks, months, or permanently,” according to Zatko’s whistleblower disclosure. (Twitter slammed Zatko and broadly defended himself against the allegations, saying the disclosure paints a “false narrative” from the company.)

News of the data center outage comes a day before Zatko is scheduled to testify before the Senate Judiciary Committee.

Twitter did not disclose the number or location of its data centers, but Zatko’s whistleblower disclosure cites a public news report that identifies the Twitter data center in Sacramento and another in Atlanta. In 2020, Amazon announced that Twitter had selected its cloud computing platform, Amazon Web Services, to serve certain tweets from Amazon data centers.

In a statement on the Sacramento outage, a Twitter spokesperson told CNN, “There have been no disruptions affecting people’s ability to access and use Twitter at this time. Our teams remain equipped with the tools and resources they need to ship updates and will continue to work to provide a seamless Twitter experience.

Data centers need “reliable water, electricity, humidity controls and refrigeration to live on,” said retired Brigadier General Greg Touhill, who served as the data center’s chief security officer. US government information in 2016 and 2017.

“You want redundancy, not duplication, of your data locations to improve your cyber resilience so you can take a beating in the event of a natural disaster [or other event] that can destroy a single piece of equipment or a data center,” Touhill, who now leads the CERT division of the Software Engineering Institute at Carnegie Mellon University, told CNN.