We are excited to announce the immediate general availability of Instaclustr for Apache Kafka® and Apache Kafka® Connect 3.6.1 on the Instaclustr Managed Platform! With this release, we’re also moving KRaft mode for Kafka to general availability. 

When creating a Kafka cluster using Kafka 3.6.1, our customers now have the option to choose whether they would like the cluster to be provisioned in KRaft mode or in Apache ZooKeeper™ mode.  

While customers have had this option since we released Kafka version 3.3.1, the key difference is that now with Kafka 3.6.1 we are offering KRaft as general availability as opposed to public preview. This means that our customers get the same incredible coverage from our Service Level  Agreements for a Kafka 3.6.1 cluster running with KRaft as they would with a Kafka 3.6.1 cluster running with ZooKeeper. 

Beginning with this release, when creating a Kafka cluster on our managed platform the default will be to create it with KRaft mode enabled. If you would like to continue to create clusters in Kafka 3.6.1 with ZooKeeper please ensure you make the alternative selection at creation. 

This is in line with the Kafka project marking ZooKeeper as deprecated starting with Kafka 3.5.0. Given the looming deadline for the removal of ZooKeeper support completely from Kafka (currently due in Kafka 4.0), we recommend creating Kafka clusters using KRaft mode. 

As for similarities, existing customers of our Managed Kafka will be pleased to know that just like they have the option of running ZooKeeper as collocated with Kafka or on dedicated nodes, they will have the same options with KRaft. Pricing for dedicated nodes for running KRaft will  be the same or less than equivalent nodes running dedicated ZooKeeper. 

For a comparison between ZooKeeper and KRaft, refer to our 3-part series of blogs comparing their performance: Apache Kafka KRaft Abandons the Zoo(Keeper).  

For those looking for a brief overview of KRaft and how it simplifies operations, enhances scalability, and boosts performance, this short video will talk you through it. 

More details on our Instaclustr for Apache Kafka 3.6.1 release:  

• Clusters on our managed platform being upgraded from any Kafka version older than 3.6.1 will be upgraded to Kafka 3.6.1 with ZooKeeper mode enabled. Note that clusters running the Public Preview version of Kafka 3.3.1, Kafka 3.4.1, or Kafka 3.5.1 with KRaft mode cannot be upgraded.  
• For new Kafka clusters being created with Kafka 3.6.1, at the time of cluster creation you will have the option of selecting whether you would like the cluster to be created with KRaft mode or ZooKeeper (KRaft mode will be the default for a newly created cluster unless you choose otherwise). 

The Kafka project’s 3.6.0 blog post, release notes for Kafka 3.6.0, and release notes for Kafka 3.6.1 are the definitive sources of the changes in this release. In addition, here are some of the highlights that caught our (collective) eye: 

• Kafka Tiered Storage: Tiered storage support has been released as early access in Kafka 3.6.0, meaning it is currently not recommended for use in production. The Kafka project plans on enhancing support for this feature in future releases. Instaclustr for Apache Kafka is aiming to support it after this feature is out of the early access stage. 

• Improvements Related to KRaft Support: KRaft support in Kafka has been incrementally improved and extended with successive Kafka releases. The durability of the KRaft layer was improved by adding metadata transactions with KIP-868 in Kafka 3.6.0. With this, the controller is now allowed to generate atomic transactions of records that can exceed the maximum batch size. Meanwhile, KIP-938 has added support for more metrics for measuring KRaft performance. 

• Upgrade of ZooKeeper to 3.8.3: Kafka 3.6.1 comes bundled with ZooKeeper upgraded to version 3.8.2 as part of KIP-902, and then upgraded to version 3.8.3 as part of KAFKA-15658 to fix CVE-2023-44981. (You can read our assessment of this CVE and details of its impact analysis on customers of our managed Kafka and managed ZooKeeper here) 

We will be updating the lifecycle states for our older supported Kafka versions to more restrictive lifecycle states (as per our lifecycle policy) with the release of Kafka 3.6.1 on our managed platform. More details on all Kafka versions supported on our managed platform are available here.  

We encourage you to try your preferred Kafka version in your non-production environments first to confirm compatibility with your Kafka clients before using it in production environments. If you need any help spinning up new Kafka cluster or upgrading existing managed Kafka clusters, please feel free to get in touch with us via our support website.