Support for Spark 1.3.1 in OpenStack

This morning we completed the review and merge process, so support for Spark 1.3.1 will be available in the next OpenStack major release, called Liberty. At the same time we are also deprecating support for Spark 1.0.0, that will be removed in the OpenStack “M” release next spring.

Advertisements

Spark notebooks in OpenStack

We’ve developed a beta version of Spark notebook support in our fork of OpenStack Sahara. With this version it is possible to have a development environment for Spark in a few clicks, ready to be used by a data scientist to develop his applications.

Spark Notebooks offer the same features as iPython notebooks, but with the distributed computing capabilities of Spark behind the scenes. They are programmed in Scala: code and text can be mixed, with plots and other data visualizations generated on the fly.

With our changes in Sahara, Noteboks processes can be deployed together with a Spark cluster and will be configuspark notebook screenshotred and ready to go.