Menu
An in-depth look at Facebook's new data centre network design...

An in-depth look at Facebook's new data centre network design...

While most mainstream enterprise don’t relate to the scale, budget, and skillset/personnel of an organisation like Facebook, according to Gartner analyst Andrew Lerner, there are some tangible takeaways.

For all the talk about the hyper scale data centres, there hasn’t actually been a ton of detailed information about how they do the network stuff they do.

This week, it was interesting to see some detail come out via blog/video from Facebook Network Engineer Alexey Andreyev regarding the design of the network for the new Facebook Altoona, IA data centre.

While most mainstream enterprise don’t relate to the scale, budget, and skillset/personnel of an organisation like Facebook, according to Gartner analyst Andrew Lerner, there are some tangible takeaways.

“In other words, while a lot of their design constructs are built for scale, many of these principles can apply to smaller data centre networks,” he says.

“Here are some of my key takeaways, and also food for thought the next time you “like” something or “tag” somebody…”

Keeping it Simple:

From the Facebook blog: “Our goal is to make deploying and operating our networks easier and faster over time…”

“I couldn’t agree more, and improving/simplifying Network Operations is a key area when we evaluate solutions as part of the Data Centre Networking Magic Quadrant,” Lerner adds.

“One of the specific ways Facebook simplifies things is to automate wherever they can, which reduces manual error and scales much better (here’s a related blog on network automation).”

Less is More:

Facebook uses smaller, simpler and cheaper network switching infrastructure.

According to Lerner, this has direct applicability in the mainstream, and here’s Facebook’s take on it: “…it requires only basic mid-size switches to aggregate the TORs.

"The smaller port density of the fabric switches makes their internal architecture very simple, modular, and robust, and there are several easy-to-find options available from multiple sources.“

Network Pods:

They refer to their design as a Core/Pod architecture, with a Pod containing 48 server racks, built in a leaf/spine architecture.

“Interconnectivity between pods is 40G and not oversubscribed, also in a leaf/spine architecture,” Lerner observes.

“The pod approach is modular, and allows them to evolve and iterate their network design as requirements change and technological capabilities change/advance.”

Traffic Flows:

While Lerner speculates that many network practitioners now realise that traffic is shifting from traditional North/South (app-to-user) patterns to East/West (app-to-app) the blog includes a powerful data point:

“What happens inside the Facebook data centres – “machine to machine” traffic – is several orders of magnitude larger than what goes out to the Internet.”

Similarly, Lerner says that Cisco recently reported a study that intra-data centre traffic was 77% in 2013, and will remain high through 2018.

“We’ve been seeing this trend for several years due to changing application architectures among other things,” he notes.

“Net net, we recommend that new data centre network builds should be a 1- or 2- tier Ethernet fabric that is optimised for both north/south and east/west with deterministic latency between any two points.”

Vendors?

The blog post doesn’t mention specific vendors, but Lerner reminds that Facebook has previously blogged about using disaggregated switching approaches with their own software (FBOSS) running on white-box style hardware.

Here’s some more information on the topic of disaggregation, with additional research coming soon…

But Wait, no SDN?

While there’s no explicit mention of SDN controllers per say, Lerner believes “they’re doing some interesting stuff.”

“They run L3 ECMP using BGP, but there is a centralised BGP controller with “override” capability which sounds a bit SDN-ish to me,” he explains.

Follow Us

Join the New Zealand Reseller News newsletter!

Error: Please check your email address.

Tags GartnerData CentreFacebook

Featured

Slideshows

Educating from the epicentre - Why distributors are the pulse checkers of the channel

Educating from the epicentre - Why distributors are the pulse checkers of the channel

​As the channel changes and industry voices deepen, the need for clarity and insight heightens. Market misconceptions talk of an “under pressure” distribution space, with competitors in that fateful “race for relevance” across New Zealand. Amidst the cliched assumptions however, distribution is once again showing its strength, as a force to be listened to, rather than questioned. Traditionally, the role was born out of a need for vendors and resellers to find one another, acting as a bridge between the testing lab and the marketplace. Yet despite new technologies and business approaches shaking the channel to its very core, distributors remain tied to the epicentre - providing the voice of reason amidst a seismic industry shift. In looking across both sides of the vendor and partner fences, the middle concept of the three-tier chain remains centrally placed to understand the metrics of two differing worlds, as the continual pulse checkers of the local channel. This exclusive Reseller News Roundtable, in association with Dicker Data and rhipe, examined the pivotal role of distribution in understanding the health of the channel, educating from the epicentre as the market transforms at a rapid rate.

Educating from the epicentre - Why distributors are the pulse checkers of the channel
Kiwi channel reunites as After Hours kicks off 2017

Kiwi channel reunites as After Hours kicks off 2017

After Hours made a welcome return to the channel social calendar last night, with a bumper crowd of distributors, vendors and resellers descending on The Jefferson in Auckland to kickstart 2017. Photos by Maria Stefina.

Kiwi channel reunites as After Hours kicks off 2017
Arrow exclusively introduces Tenable Network Security to A/NZ channel

Arrow exclusively introduces Tenable Network Security to A/NZ channel

Arrow Electronics introduced Tenable Network Security to local resellers in Sydney last week, officially launching the distributor's latest security partnership across Australia and New Zealand. Representing the first direct distribution agreement locally for Tenable specifically, the deal sees Arrow deliver security solutions directly to mid-market and enterprise channel partners on both sides of the Tasman.

Arrow exclusively introduces Tenable Network Security to A/NZ channel
Show Comments