Announcing the General Availability of Cloudera Streams Management

Last month, you heard us get all excited about Cloudera Stream Processing - our solution to enabling all of Cloudera customers get the benefits of having the latest, secure version of Apache Kafka along with Schema Registry and Kafka Streams. We are proud to have been supporting hundreds of active customers with their Kafka implementations and now, even more eager to serve several hundred more customers.

As much as Kafka is being used as the core stream processing engine across most of the streaming architectures today across enterprises, it also brings about its own set of challenges within platform operations, DevOps and the security and governance teams. Vanilla Kafka does not have any management interface out of the box to help such teams with visibility. On a related note, business continuity is the #1 priority for IT organizations when it comes to such real-time streaming architectures. If the primary Kafka cluster goes down, all eyes are on IT and DataOps teams to see if they have devised an appropriate DR (Disaster Recovery) plan to bring up an alternate cluster to keep the business functioning smoothly. Again, no out-of-the-box tools are available for effective HA (High availability)/DR management within Kafka.

In order to address such challenges, Cloudera is delighted to introduce the management and monitoring bundle for Kafka – Cloudera Streams Management (CSM). CSM is primarily made up of two products –
1. Cloudera Streams Messaging Manager (SMM) – This is the management/monitoring dashboard for Kafka that has been hugely popular since last year. It has been revolutionizing the way enterprises have been gaining visibility into their Kafka clusters. We love that our customers are raving about it!
2. Cloudera Streams Replication Manager (SRM) – This is a brand new product that we are introducing under the umbrella of CSM. Kafka replication has been a fundamental challenge for enterprises for either HA or DR needs. Inherent Kafka tooling such as Mirrormaker was not built for replication and so, our customers asked for something simple, intuitive and manageable. SRM helps with your business continuity by enabling Kafka replication across multiple clusters in your environment.

What’s New?
Streams Messaging Manager (SMM)
We identified “Kafka blindness” amongst our top 30 Kafka customers a couple of years ago. We set forth to come up with an innovative solution that has been highly praised by one and all for its intuitive design and usability. With this release of CSM, we have integrated SMM with Cloudera Manager for service management, Apache Sentry for role-based authorization and Schema Registry service that provides governance and schema management capabilities. This set of features truly enables all of Cloudera customers to take advantage of SMM within the context of a familiar environment such as Cloudera Manager. In this release, we have also added a new DevOps monitoring capability with end-to-end latency view across producers to topics/brokers to consumers. This can help answer questions like “What is the average latency of messages consumed after being produced?”

Streams Replication Manager (SRM)
Apache Kafka supports internal replication to support data availability within a cluster. However, enterprises require that the data availability and durability guarantees span across entire clusters and site failures. Replication of data across clusters and sites is critical for a lot of the use cases including DR, aggregation for analytics, access isolation, geo-proximity, cloud migration and for legal and compliance reasons. Mirrormaker is a utility that is often used today for Kafka replication needs. However, Mirrormaker has severe limitations for enterprises looking to do mission-critical data replication between their Kafka clusters.

So, we led the proposal for creating Kafka Mirrormaker 2, which addresses the limitations of Mirrormaker 1 with the ability to dynamically change configurations, keep the topic properties in sync across clusters and improve performance significantly. Moreover, handling active-active clusters and disaster recovery are use cases that MM2 supports out-of-the-box.

Streams Replication Manager is a Cloudera open-source innovation and is based on MM2. SRM  provides enterprises with the ability to implement cross-cluster Kafka topic replication. This enables enterprises to implement use cases with Kafka which require disaster tolerant architectures or other complex architectures such as active/active cluster setups. SRM is also tightly integrated with the SMM user interface such that there is a new Cluster Replications view within SMM. This provides a monitoring view for replication flows happening in the SRM cluster. The view shows replication details with metrics at a cluster and topic level about the list of topics being replication, the status of each replication, throughput between the clusters and any replication latency as well. The replication cluster view in SMM also allows you to create alerts on replication metrics.

Cloudera is committed to focusing on the streaming market and in enhancing our already comprehensive Kafka offerings. Stay tuned with us for more exciting announcements coming your way!

Dinesh Chandrasekhar is Head of Product Marketing at Cloudera.