Menu
Microsoft brings .NET dev to Apache Spark

Microsoft brings .NET dev to Apache Spark

.NET for Apache Spark 1.0 provides high-performance .NET APIs to Apache Spark including Spark SQL, Spark Streaming, and MLlib

Credit: Dreamstime

Microsoft and the .NET Foundation have released version 1.0 of .NET for Apache Spark, an open source package that brings .NET development to the Spark analytics engine for large-scale data processing.

Announced October 27, .NET for Apache Spark 1.0 has support for .NET applications targeting .NET Standard 2.0 or later. Users can access Spark DataFrame APIs, write Spark SQL, and create user-defined functions UDFs).

The .NET for Apache Spark framework is available on the .NET Foundation’s GitHub page or from NuGet. Other capabilities of .NET for Apache Spark 1.0 include an API extension framework to add support for additional Spark libraries including Linux Foundation Delta Lake, Microsoft OSS Hyperspace, ML.NET, and Apache Spark MLlib functionality.

This is in addition to .NET for Apache Spark programs that are not UDFs show the same speed as Scala and PySpark-based non-UDF applications. If applications include UDFs, .NET for Apache Spark programs are at least as fast as PySpark programs or might be faster.

Furthermore, .NET for Apache Spark is built into Azure Synapse and Azure HDInsight. It also can be used in other Apache Spark cloud offerings including Azure Databricks.

The first public version of the project was announced in April 2019. Driving the development of .NET for Apache Spark was increased demand for an easier way to build big data applications instead of having to learn Scala or Python. The project is operated under the .NET Foundation and has been filed as a Spark Project Improvement Proposal to be considered for inclusion in the Apache Spark project directly.

Looking ahead, Microsoft is addressing obstacles including setting up prerequisites and dependencies and finding quality documentation, with examples such as community-contributed “ready-to-run” Docker images and updates to .NET for Apache Spark documentation.

Another priority is supporting deployment options including integration with CI/CD devops pipelines and publishing jobs directly from Visual Studio.


Follow Us

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Tags Microsoft

Featured

Slideshows

The Kiwi channel gathers for the 2020 Reseller News Women in ICT Awards

The Kiwi channel gathers for the 2020 Reseller News Women in ICT Awards

Hundreds of leaders from the New Zealand IT industry gathered at the Hilton in Auckland on 17 November to celebrate the finest female talent in the Kiwi channel and recognise the winners of the Reseller News Women in ICT Awards (WIICTA) 2020.

The Kiwi channel gathers for the 2020 Reseller News Women in ICT Awards
Leading female front runners honoured at the 2020 Reseller News Women in ICT Awards

Leading female front runners honoured at the 2020 Reseller News Women in ICT Awards

The leading female front runners of the New Zealand ICT industry joined together for the annual Reseller News Women in ICT Awards event at the Hilton in Auckland, during which hundreds of guests celebrated 13 outstanding individuals who won awards, chosen from more than 50 finalists representing over 30 organisations.

Leading female front runners honoured at the 2020 Reseller News Women in ICT Awards
Channel gathers to celebrate the Reseller News Innovation Awards 2020 winners

Channel gathers to celebrate the Reseller News Innovation Awards 2020 winners

More than 500 channel leaders gathered in Auckland on 21 October at the ​Reseller News Innovation Awards ​2020 to celebrate the achievements of the New Zealand technology industry's top partners, start-ups, vendors, distributors and individuals.

Channel gathers to celebrate the Reseller News Innovation Awards 2020 winners
Show Comments