Menu
Apache Spark 3.0 adds Nvidia GPU support for machine learning

Apache Spark 3.0 adds Nvidia GPU support for machine learning

The next major release of the in-memory data processing framework will support GPU-accelerated functions courtesy of Nvidia RAPIDS

Credit: Dreamstime

Apache Spark, the in-memory big data processing framework, will become fully GPU accelerated in its soon-to-be-released 3.0 incarnation. Best of all, today’s Spark applications can take advantage of the GPU acceleration without modification; existing Spark APIs all work as-is.

The GPU acceleration components, provided by Nvidia, are designed to complement all phases of Spark applications including ETL operations, machine learning training, and inference serving.

Nvidia’s Spark contributions draw on the RAPIDS suite of GPU-accelerated data science libraries. Many of RAPIDS’ internal data structures, like dataframes, complement Spark’s own, but getting Spark to use RAPIDS natively has taken nearly four years of work.

Spark 3.0 speedups don’t come solely from GPU acceleration. Spark 3.0 also reaps performance gains by minimizing data movement to and from GPUs. When data does need to be moved across a cluster, the Unified Communication X framework shuttles it directly from one block of GPU memory to another with minimal overhead.

According to Nvidia, a preview release of Spark 3.0 running on the Databricks platform yielded a seven-fold performance improvement when using GPU acceleration, though details about the workload and its dataset were not available. 

No firm date has been given for general availability of Spark 3.0. You can download preview releases from the Apache Spark project website.


Follow Us

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Featured

Slideshows

The Kiwi channel gathers for the 2020 Reseller News Women in ICT Awards

The Kiwi channel gathers for the 2020 Reseller News Women in ICT Awards

Hundreds of leaders from the New Zealand IT industry gathered at the Hilton in Auckland on 17 November to celebrate the finest female talent in the Kiwi channel and recognise the winners of the Reseller News Women in ICT Awards (WIICTA) 2020.

The Kiwi channel gathers for the 2020 Reseller News Women in ICT Awards
Leading female front runners honoured at the 2020 Reseller News Women in ICT Awards

Leading female front runners honoured at the 2020 Reseller News Women in ICT Awards

The leading female front runners of the New Zealand ICT industry joined together for the annual Reseller News Women in ICT Awards event at the Hilton in Auckland, during which hundreds of guests celebrated 13 outstanding individuals who won awards, chosen from more than 50 finalists representing over 30 organisations.

Leading female front runners honoured at the 2020 Reseller News Women in ICT Awards
Channel gathers to celebrate the Reseller News Innovation Awards 2020 winners

Channel gathers to celebrate the Reseller News Innovation Awards 2020 winners

More than 500 channel leaders gathered in Auckland on 21 October at the ​Reseller News Innovation Awards ​2020 to celebrate the achievements of the New Zealand technology industry's top partners, start-ups, vendors, distributors and individuals.

Channel gathers to celebrate the Reseller News Innovation Awards 2020 winners
Show Comments