CTO Matei Zaharia from Databricks on the future of Spark

During the Spark + AI Summit in Amsterdam, October 15-17th, BI-Platform had a conversation with Matei Zaharia, Chief Technical Officer at Databricks and developer of Apache Spark. At the Summit Databricks announced Model Registry and Delta Lake being hosted by the Linux Foundation. Also Databricks announced an investment from about 100 million euro in an European Development Center in Amsterdam.

In this interview Matei Zaharia talks with BI-Platform about the relationship between Spark and Hadoop, developments on the issue of Python API’s like Koalas, data placeholders, deploying Spark within containers, the upcoming Spark 3.0, model management within MLflow with the new Model Registry component and the future of the Spark core libraries.

All our video interviews by BI-Platform will also be published on our YouTube channel. Stay tuned and click here to subscribe.