Menu
How Netflix survived the Amazon EC2 reboot

How Netflix survived the Amazon EC2 reboot

The video streaming service was able to stay online even as its cloud hosting provider rebooted its servers

Sometimes the best path to success is to learn how to avoid failure.

Netflix was able to keep serving its customers while its cloud hosting provider, Amazon Web Services (AWS), rebooted servers, because it had prepared for that happening.

"When we got the news about the emergency EC2 [Elastic Cloud Compute] reboots, our jaws dropped. When we got the list of how many Cassandra nodes would be affected, I felt ill," said Christos Kalantzis, Netflix engineering manager of cloud database engineering, in a Netflix blog post discussing the outage.

Amazon announced to EC2 customers on Sept. 25 that it would be updating its servers and that a small percentage would require a reboot, which could potentially disrupt customer services. AWS did not specify which of their virtual hosts would be rebooted or when. It was revealed later that AWS was fixing a vulnerability in the Xen hypervisor, which underpins EC2.

Netflix is one of Amazon's largest customers. And its 50 million customers expect to be able to stream TV shows, movies and other content at any time. If Netflix wasn't prepared to mitigate potential outages, the company -- and not Amazon -- would have a lot of angry customers.

But Netflix had architected its service to be resilient, so that if one Amazon data center went down, operations could be switched over to another with barely a noticeable bump to customers. It also looked for ways to minimize downtime that occurred when its services did need to be rebooted.

The company even went the extra mile and aggressively looked for ways to try to disrupt its own services through a set of tools called the Simian Army that are designed to periodically and randomly kill Netflix services. The thinking goes that any Netflix service should be resilient enough to keep running through an attack from one such tool. If it isn't, then the Netflix engineers redesign the service to make it more reliable.

Even with its systems hardened by abuse from Chaos Monkey and other Simian Army tools, the engineers were still worried about the AWS reboot.

In particular, concern centered around the 2,700 Cassandra databases that the company runs on AWS.

Databases, as the blog post pointed out, are "the pampered and spoiled princes of the application world." They are run on the best hardware, get lots of attention from database engineers and still can be fussy creatures.

Netflix deliberately chose to use the Cassandra database over more traditional choices such as Oracle's databases because, as a NoSQL database, Cassandra could be spread across multiple servers in such a way that if one of the nodes failed, the database could keep running. Over the past year, the company had been subjecting Cassandra to Chaos Monkey testing, with promising results.

The AWS reboot would be the first true test of Cassandra's reliability, however. The entire cloud database engineering team was on alert.

In the end, and thanks to Chaos Monkey testing, most all of the Cassandra nodes remained online. Of the 218 Cassandra nodes that were rebooted, only 22 did not return to a full operational state, and those were successfully restarted with minimal human intervention.

"Repeatedly and regularly exercising failure, even in the persistence layer, should be part of every company's resilience planning," the blog concluded. "If it wasn't for Cassandra's participation in Chaos Monkey, this story would have ended much differently."

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

Follow Us

Join the New Zealand Reseller News newsletter!

Error: Please check your email address.

Tags amazoncloud computingnetflixinternetInfrastructure services

Slideshows

Top 50 defining moments of the New Zealand channel in 2016

Top 50 defining moments of the New Zealand channel in 2016

Reseller News looks back on a tumultuous 12 months for the New Zealand channel, assessing the fallout from a year of sizeable industry change. Whether it be local or global mergers and acquisitions, distribution deals or job changes, the channel that started the year differs somewhat to the one set to finish it - Reseller News assesses the key moments that made 2016.​

Top 50 defining moments of the New Zealand channel in 2016
​Hewlett Packard Enterprise honours high achieving NZ channel

​Hewlett Packard Enterprise honours high achieving NZ channel

Hewlett Packard Enterprise honoured its top performing Kiwi partners at the second running of its HPE Partner Awards in New Zealand, held at a glitzy ceremony in Auckland. Recognising excellence across eight categories - from distributors to resellers - the tech giant celebrated its first year as a standalone company, following its official split from HP in 2015.

​Hewlett Packard Enterprise honours high achieving NZ channel
Nutanix treats channel partners to Christmas cruise

Nutanix treats channel partners to Christmas cruise

Nutanix recently took to the seas for a Christmas Cruise around Sydney Harbour with its Australia and New Zealand staff, customers and partners to celebrate a stellar year for the vendor. With the sun out, they were all smiles and mingled over drinks and food.

Nutanix treats channel partners to Christmas cruise
Show Comments