Menu
SC13: Elevation plays a role in memory error rates

SC13: Elevation plays a role in memory error rates

A study by AMD and the Department of Energy showed a higher supercomputer had more memory problems

A study from AMD and the Department of Energy showed how SRAM in the Cielo supercomputer had more transient errors than those in the Jaguar supercomputer, probably due to the difference in elevation between the two supercomputers

A study from AMD and the Department of Energy showed how SRAM in the Cielo supercomputer had more transient errors than those in the Jaguar supercomputer, probably due to the difference in elevation between the two supercomputers

With memory, as with real estate, location matters. A group of researchers from Advanced Micro Devices (AMD) and the Department of Energy's Los Alamos National Laboratory have found that the altitude at which SRAM (static random access memory) resides can influence how many random errors the memory produces.

In a field study of two high-performance computers, the researchers found that L2 and L3 caches had more transient errors on the supercomputer located at a higher altitude, compared with the one closer to sea level. They attributed the disparity largely to lower air pressure and higher cosmic ray-induced neutron strikes.

Strangely, higher elevation even led to more errors within a rack of servers, the researchers found. Their tests showed that memory modules on the top of a server rack had 20 percent more transient errors than those closer to the bottom of the rack. However, it's not clear what causes this smaller-scale effect.

Vilas Sridharan, an AMD technical staff member, presented the findings Thursday at the SC13 supercomputing conference, being held this week in the mile-high city of Denver.

Using the error logs of two large high-performance computers, the study examined the characteristics of transient memory errors, in which a memory module may store a 1 as a 0, or vice versa.

Transient errors are different from permanent or even intermittent errors, which are usually caused by hardware failure, Sridharan said. Transient errors appear more randomly and are not usually the fault of machinery. They are relatively rare, but depending on where they occur, they can cause a cascade of additional system errors.

The group studied the monthly transient fault rates of SRAM--the L2 and L3 caches within processors--in two large Cray supercomputers, each running thousands of AMD processors.

One supercomputer was the Jaguar system at Oak Ridge National Laboratory in Oak Ridge, Tennessee, which is approximately 817 feet (249 meters) above sea level, according to an online altitude finder.

The other system under study was the Cielo supercomputer at the Los Alamos National Laboratory in Los Alamos, New Mexico, which is about 7,058 feet (2,151 meters) above sea level.

The group had found that, when all other possible confounding issues were factored out, Cielo's SRAM had a "significantly higher rate of SRAM faults," compared with Jaguar's SRAM, Sridharan said.

For example, with L3 caches, Cielo was bedeviled by 735 transient faults for every 219 that Jaguar endured. L2 transient faults across the two machines showed a similar relationship.

The findings were not a surprise, according to Sridharan. It has long been theorized that transient memory errors can come from the high-energy impact of neutrons from cosmic rays, which is more pronounced at higher elevations. Other factors related to elevation, such as air pressure, may also play a role.

"This is theoretically well-known, but it is nice to see the data," Sridharan said.

Another effect was slightly more mysterious: The SRAM at the top of server racks had a significantly greater number of transient errors than that at the middle or the bottom of the same racks, within both Jaguar and Cielo.

"There is a trend towards a higher rate of SRAM faults as you go up the rack," Sridharan said. "This is something we don't really have a good explanation for."

SRAM on the server on the top of the rack had 20 percent more transient errors than the SRAM on the servers on the lower levels. "This is not a huge effect, but it is a consistent one," Sridharan said.

The difference probably could not be attributed solely to cosmic rays, Sridharan said. He briefly speculated on a number of possible causes. For example, because heat rises, the servers at the top of a rack are hotter than those on the bottom. Heat is a well-known culprit in equipment failure.

A low-cost solution, such as installing heat shielding on server racks, may be worth investigating, Sridharan said.

In the study, the group also looked at the DRAM memory faults. They examined memory from three different vendors and found that the fault rate of one vendor was four times the rate of another vendor. The group did not release the names of the vendors but did alert the vendor with the leading error rate about the comparatively high rate of faults for its products.

Joab Jackson covers enterprise software and general technology breaking news for The IDG News Service. Follow Joab on Twitter at @Joab_Jackson. Joab's e-mail address is Joab_Jackson@idg.com

Follow Us

Join the New Zealand Reseller News newsletter!

Error: Please check your email address.

Tags U.S. Department of Energypopular scienceAdvanced Micro DevicesComponentsmemory

Featured

Slideshows

Arrow exclusively introduces Tenable Network Security to A/NZ channel

Arrow exclusively introduces Tenable Network Security to A/NZ channel

Arrow Electronics introduced Tenable Network Security to local resellers in Sydney last week, officially launching the distributor's latest security partnership across Australia and New Zealand. Representing the first direct distribution agreement locally for Tenable specifically, the deal sees Arrow deliver security solutions directly to mid-market and enterprise channel partners on both sides of the Tasman.

Arrow exclusively introduces Tenable Network Security to A/NZ channel
Examining the changing job scene in the Kiwi channel

Examining the changing job scene in the Kiwi channel

Typically, the New Year brings new opportunities for personnel within the Kiwi channel. 2017 started no differently, with a host of appointments, departures and reshuffles across vendor, distributor and reseller businesses. As a result, the job scene across New Zealand has changed - here’s a run down of who is working where in the year ahead…

Examining the changing job scene in the Kiwi channel
​What are the top 10 tech trends for New Zealand in 2017?

​What are the top 10 tech trends for New Zealand in 2017?

Digital Transformation (DX) has been a critical topic for business over the last few years and IDC is now predicting a step change as DX reaches macroeconomic levels. By 2020 a DX economy will emerge and it will become the core of what New Zealand industries focus on. From the board level through to the C-Suite, Kiwi organisations must be prepared to think and act digital when the DX economy emerges in 2017.

​What are the top 10 tech trends for New Zealand in 2017?
Show Comments