Menu
Apache Spark 3.0 adds Nvidia GPU support for machine learning

Apache Spark 3.0 adds Nvidia GPU support for machine learning

The next major release of the in-memory data processing framework will support GPU-accelerated functions courtesy of Nvidia RAPIDS

Credit: Dreamstime

Apache Spark, the in-memory big data processing framework, will become fully GPU accelerated in its soon-to-be-released 3.0 incarnation. Best of all, today’s Spark applications can take advantage of the GPU acceleration without modification; existing Spark APIs all work as-is.

The GPU acceleration components, provided by Nvidia, are designed to complement all phases of Spark applications including ETL operations, machine learning training, and inference serving.

Nvidia’s Spark contributions draw on the RAPIDS suite of GPU-accelerated data science libraries. Many of RAPIDS’ internal data structures, like dataframes, complement Spark’s own, but getting Spark to use RAPIDS natively has taken nearly four years of work.

Spark 3.0 speedups don’t come solely from GPU acceleration. Spark 3.0 also reaps performance gains by minimizing data movement to and from GPUs. When data does need to be moved across a cluster, the Unified Communication X framework shuttles it directly from one block of GPU memory to another with minimal overhead.

According to Nvidia, a preview release of Spark 3.0 running on the Databricks platform yielded a seven-fold performance improvement when using GPU acceleration, though details about the workload and its dataset were not available. 

No firm date has been given for general availability of Spark 3.0. You can download preview releases from the Apache Spark project website.


Follow Us

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Events

Featured

Slideshows

Meet the Reseller News 30 Under 30 Tech Awards 2020 winners

Meet the Reseller News 30 Under 30 Tech Awards 2020 winners

This year’s Reseller News 30 Under 30 Tech Awards were held as an integral part of the first entirely virtual Emerging Leaders​ forum, an annual event dedicated to identifying, educating and showcasing the New Zealand technology market’s rising stars. The 30 Under 30 Tech Awards 2020 recognised the outstanding achievements and business excellence of 30 talented individuals​, across both young leaders and those just starting out. In this slideshow, Reseller News honours this year's winners and captures their thoughts about how their ideas of leadership have changed over time.​

Meet the Reseller News 30 Under 30 Tech Awards 2020 winners
Reseller News Exchange Auckland: Beyond the myths — how partners can master cloud security

Reseller News Exchange Auckland: Beyond the myths — how partners can master cloud security

This exclusive Reseller News Exchange event in Auckland explored the challenges facing the partner community on the cloud security frontier, as well as market trends, customer priorities and how the channel can capitalise on the opportunities available. In association with Arrow, Bitdefender, Exclusive Networks, Fortinet and Palo Alto Networks. Photos by Gino Demeer.

Reseller News Exchange Auckland: Beyond the myths — how partners can master cloud security
Reseller News welcomes industry figures at 2020 Hall of Fame lunch

Reseller News welcomes industry figures at 2020 Hall of Fame lunch

Reseller News welcomed 2019 inductees - Leanne Buer, Ross Jenkins and Terry Dunn - to the fourth running of the Reseller News Hall of Fame lunch, held at the French Cafe in Auckland. The inductees discussed the changing face of the IT channel ecosystem in New Zealand and what it means to be a Reseller News Hall of Fame inductee. Photos by Gino Demeer.

Reseller News welcomes industry figures at 2020 Hall of Fame lunch
Show Comments