Microsoft launches Spark for Azure HDInsight and pushes into consumer AI

By:  Jamie Davies

Almost 12 months after releasing the public preview of Spark for Azure HDInsight Microsoft has announced general availability of the proposition to the industry, as well as extending Cortana’s offering to the Xbox.

Making the announcement on the Azure blog, Oliver Chiu, Product Marketing Manager for Hadoop/Big Data and Data Warehousing, outlined the improvements made on the offering as well as the company’s efforts to make big data easy and more approachable. The company claims the Hadoop and Spark cloud service is now an enterprise-ready solution which is fully managed, secured, and highly available.

“Since we announced the public preview, Spark for HDInsight has gained rapid adoption and is now 50% of all new HDInsight clusters deployed,” said Chiu. “With GA (General Availability), we are revealing improvements we’ve made to the service to make Spark hardened for the enterprise and easy for your users. This includes improvements to the availability, scalability, and productivity of our managed Spark service.”

Features for the new services include new capabilities to the YARN resource manager to create an open source Apache licensed REST web service, as well as integrations between Spark and Azure Data Lake Store to increase scalability, Spark and Data Lake Store to increase security, Jupyter (iPython) notebooks and Power BI to build interactive visualizations over data of any size.

Alongside the Spark announcement, the Microsoft team has also ventured into the consumer AI market as Cortana will now be available on Xbox One, Xbox Live and Windows Stores. Starting with a limited Xbox Preview audience for Xbox One, the offering will be more widely released to the Xbox app (beta) on Windows 10 over the course of the summer.

Starting out with U.S., U.K., France, Italy, Germany and Spain, Cortana voice commands on Xbox One will work with both headsets and Kinect. Firstly users will be able to find new games, see what your friends are up to, start a party and accomplish common tasks, though new features will be adding over time.

Despite the company suffering a very public set-back in the AI world with the malfunction of Tay, the team have pressed forward, seemingly prioritizing AI for new features throughout the portfolio. CEO Satya Nadella stated at the Microsoft Build 2016 event AI would feature heavily in future investments, as the team target a Conversation-as-a-Platform proposition. The Cortana Intelligence Suite, which was launched at the event, allows developers to build apps and bots which interact with customers in a personalized way, but also react to real-world developments in real-time.

“As an industry, we are on the cusp of a new frontier that pairs the power of natural human language with advanced machine intelligence,” said Nadella. “At Microsoft, we call this Conversation-as-a-Platform, and it builds on and extends the power of the Microsoft Azure, Office 365 and Windows platforms to empower developers everywhere.”