Menu
Microsoft brings .NET dev to Apache Spark

Microsoft brings .NET dev to Apache Spark

.NET for Apache Spark 1.0 provides high-performance .NET APIs to Apache Spark including Spark SQL, Spark Streaming, and MLlib

Credit: Dreamstime

Microsoft and the .NET Foundation have released version 1.0 of .NET for Apache Spark, an open source package that brings .NET development to the Spark analytics engine for large-scale data processing.

Announced October 27, .NET for Apache Spark 1.0 has support for .NET applications targeting .NET Standard 2.0 or later. Users can access Spark DataFrame APIs, write Spark SQL, and create user-defined functions UDFs).

The .NET for Apache Spark framework is available on the .NET Foundation’s GitHub page or from NuGet. Other capabilities of .NET for Apache Spark 1.0 include an API extension framework to add support for additional Spark libraries including Linux Foundation Delta Lake, Microsoft OSS Hyperspace, ML.NET, and Apache Spark MLlib functionality.

This is in addition to .NET for Apache Spark programs that are not UDFs show the same speed as Scala and PySpark-based non-UDF applications. If applications include UDFs, .NET for Apache Spark programs are at least as fast as PySpark programs or might be faster.

Furthermore, .NET for Apache Spark is built into Azure Synapse and Azure HDInsight. It also can be used in other Apache Spark cloud offerings including Azure Databricks.

The first public version of the project was announced in April 2019. Driving the development of .NET for Apache Spark was increased demand for an easier way to build big data applications instead of having to learn Scala or Python. The project is operated under the .NET Foundation and has been filed as a Spark Project Improvement Proposal to be considered for inclusion in the Apache Spark project directly.

Looking ahead, Microsoft is addressing obstacles including setting up prerequisites and dependencies and finding quality documentation, with examples such as community-contributed “ready-to-run” Docker images and updates to .NET for Apache Spark documentation.

Another priority is supporting deployment options including integration with CI/CD devops pipelines and publishing jobs directly from Visual Studio.


Follow Us

Join the newsletter!

Or

Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Tags Microsoft

Events

Featured

Slideshows

Channel kicks 2021 into gear as After Hours returns to Auckland

Channel kicks 2021 into gear as After Hours returns to Auckland

After Hours made a welcome return to the channel social calendar with a bumper crowd of partners, distributors and vendors descending on The Pantry at Park Hyatt in Auckland to kick-start 2021.

Channel kicks 2021 into gear as After Hours returns to Auckland
The Kiwi channel gathers for the 2020 Reseller News Women in ICT Awards

The Kiwi channel gathers for the 2020 Reseller News Women in ICT Awards

Hundreds of leaders from the New Zealand IT industry gathered at the Hilton in Auckland on 17 November to celebrate the finest female talent in the Kiwi channel and recognise the winners of the Reseller News Women in ICT Awards (WIICTA) 2020.

The Kiwi channel gathers for the 2020 Reseller News Women in ICT Awards
Show Comments