Instaclustr for Apache Kafka® 3.8.1 and Kafka Tiered Storage Are Now Generally Available

February 11, 2025 | By Varun Ghai

NetApp is excited to announce the General Availability of Instaclustr for Apache Kafka® 3.8.1 as well as Tiered Storage and Apache Kafka® Connect 3.8.1 on the Instaclustr Managed Platform.  

Kafka 3.8.1 is the first bugfix release for Kafka 3.8.0 and the first Kafka 3.8.x to be released on our managed platform. Compared to the previous minor Kafka release, these bring fixes for multiple smaller issues and security vulnerabilities related to components and libraries used by Kafka. A couple of the ones that caught our collective eye: 

• KAFKA-15420: Kafka Tiered Storage V1 work, which included a group of about 30 tasks was marked fully complete with Kafka 3.8.1, with many of listed features and fixes having been incrementally included in previous Kafka versions. 

• KIP-390: Introduces a mechanism for allowing the compression level used by Kafka to be configured. Previously, this was pre-selected. Tests conducted as part of the KIP’s implementation showed improvements in the producer’s messages/sec rate of up to 150%. Instaclustr for Kafka clusters are already configured with Kafka defaults, so if you wish to use the new compression level, specifying the needed values on the producer side will ensure the brokers use those without needing any further configuration changes. 

• KIP-848: Provides in Preview the next generation of the consumer rebalance protocol, available only for KRaft mode clusters. While moving complexity away from the consumer to the Group Coordinator, an additional goal of this KIP is that a consumer should not be impacted at all by a rebalance if its assignment is not changed. Note that as a feature still in Preview, we do not recommend its use in a production environment. If you’d like to test it in a non-production environment, please reach out to our support team to get started. 

• KAFKA-16347: ZooKeeper bumped to version 3.8.4.
  

With the above inclusions, we’re thrilled to announce Tiered Storage for Kafka as a GA feature with Kafka 3.8.1. We had originally released this feature starting with 3.6.1 when we announced its availability in Preview. For those keen on diving into the details of Tiered Storage, how it works, pros and cons, we’ve got a series of blogs to get you going. Additional information to help you understand how Tiered Storage in Kafka works is also available in our support pages, while steps on how to enable it for your Instaclustr for Apache Kafka cluster are provided here.) 

Tiered Storage is offered as an Enterprise Add-On. Once Enterprise Pricing (a 20% premium) is applied to your cluster, all enterprise features such as PCI or PrivateLink are available to you at no extra cost. Please contact your Customer Success Representative for more details around pricing applicable to you. 

As with previous release, we are continuing to monitor the evolution of support for migration of ZooKeeper to KRaft. With 3.8.1, we recommend the use of KRaft only on new Kafka clusters, i.e. not a migration from ZooKeeper to KRaft. In Kafka 3.9.0, which was recently released by the project, the project states support for ZooKeeper to KRaft migration to be “…the final and best iteration of our ZK migration feature”. We are validating this and aim to add support for ZooKeeper to KRaft migrations when we make Kafka 3.9.x available on our managed platform. 

With this new release, we have reviewed and updated the lifecycle states for our older supported Kafka versions (as per our lifecycle policy). To ensure you get the full benefit of our support and SLAs, please reach out to us and upgrade to a GA version. To stay on a supported version of Kafka, we advise customers upgrade their Kafka versions at least once a year to the latest GA version. More details on all Kafka versions supported on our managed platform are available here.  

Prior to deploying a new Kafka version in production, we recommend trying your preferred Kafka version in a non-production environment to confirm compatibility with your Kafka clients. If you need any help spinning up a new Kafka cluster or upgrading existing managed Kafka clusters, please feel free to get in touch with us via our support website.