Introducing Cloudera Stream Processing

Today’s streaming architectures are far more demanding in terms of scale, volume and the urgency for real-time insights. The advent of IoT and other edge related use cases have sprouted a need for a strong backbone in the enterprise architecture that can handle the differing data inflows, outflows and consumption patterns of various applications across the streaming landscape. Over the last few years, Apache Kafka has emerged as that backbone. Kafka has evolved significantly from being just a messaging platform into a complex ecosystem of components that work tightly to deliver stream processing and analytics.

Cloudera and Hortonworks have been supporting Kafka over the years and now with the combined offering under Cloudera DataFlow, we have the largest number of Kafka customers supported across the world. We have also been innovating across the Kafka ecosystem of components. The key differentiator with the way we enable our customers with Kafka is offering it as a part of the Cloudera DataFlow (CDF) platform. It is tightly integrated with the rest of the components in the platform as well as with the enterprise services for unified security and governance.

Cloudera Stream Processing (CSP) is a product within the Cloudera DataFlow platform that packs Kafka along with some key streaming components that empower enterprises to handle some of the most complex and sophisticated streaming use cases. CSP provides advanced messaging, real-time processing and analytics on real-time streaming data using Apache Kafka. CSP also supports key management and monitoring capabilities powered by Cloudera Streams Management (CSM).

Cloudera Stream Processing consists of the following:
• Support for latest version of Kafka
• Support for Kafka Streams
• Schema Registry that provides a centralized registry to manage schema for Apache Kafka and Apache NiFi
• Support for Kafka replication with Mirrormaker
• Cluster Management and monitoring with Cloudera Manager
• Rich access control with Apache Sentry

What’s coming next
We will be shortly announcing Cloudera Streams Management that includes some of the innovative management and monitoring components for enterprise Kafka –
• Streams Messaging Manager (SMM) – A single management & monitoring tool that cures “Kafka Blindness” by providing complete transparency across all aspects of Kafka – Producers, Consumers, Topics and Brokers. It enables developers/DevOps teams, Platform Operations teams and Security/Governance teams with edge-to-enterprise visibility of all that is happening within your Kafka clusters. This is a tool that has already been widely adopted by the legacy Hortonworks customer base that have Kafka. Now, soon, all legacy Cloudera customers that have Kafka can also take advantage of this tool.
• Streams Replication Manager (SRM) – This is our upcoming product on Kafka replication based on the completely re-energized Mirrormaker 2.0. Watch this space for more details on this in the coming weeks.

Dinesh Chandrasekhar is Head of Product Marketing at Cloudera.