Instaclustr for Apache Kafka® and Apache Kafka® Connect 3.7.1 Are Now Generally Available

July 29, 2024 | By Varun Ghai

NetApp is pleased to announce the general availability of Instaclustr for Apache Kafka® 3.7.1 and Apache Kafka® Connect 3.7.1 on the Instaclustr Managed Platform.  

Kafka 3.7.1 is the first bugfix release for Kafka 3.7.0 and the first Kafka 3.7.x to be released on our managed platform. The 3.7.x releases have fixed a number of several smaller issues and security vulnerabilities related to components and libraries used by Kafka. Some other noteworthy changes these releases introduce: 

• KIP-580 – Similar to KIP-144, which all the way back in Kafka 0.11.0 introduced exponential backoff for broker reconnect attempts, KIP-580 changes the currently static (100ms) backoff by implementing an exponential backoff policy for Kafka clients trying to fetch metadata. This should help when metadata request to brokers fail, by reducing the pressure and allowing metadata convergence to not be slowed down by non-serviceable repetitive fetch metadata requests from client. 

• KIP-951 – Introduces a change which minimizes the time taken to discover a new leader, and provide that information as soon as possible on the client helping in situations such as cluster rolls, reassignments, or preferred leader election. 

• KIP-980 – Allows creating Connectors in a stopped or paused state, which wasn’t possible earlier with Kafka Connect. 

• KAFKA-15046 – Resolves issues with producer latencies rising under high disk load. 

Improvements and fixes related to KRaft: 

• KIP-979 – Supports a way to independently stop KRaft processes where KRaft is running co-located with Kafka. 

• KIP-919 – In certain situations, it is beneficial for Kafka clients to be able to directly interact with KRaft controllers. This support is added as part of this KIP. 

• KAFKA-16814 – Fixes an issues with KRaft controller not starting up when partition.metadata is missing. 

• KAFKA-16886 – Fixes a bug to resolve KRaft partition reassignment failures after upgrade to 3.7.0. 

Fixes related to support of Tiered Storage: 

• KAFKA-15047 – For tiered storage enabled clusters, roll the active segment in case remote storage is enabled so that it can be eligible to be copied by the remote storage subsystem and eventually picked up for retention cleanup. 

• KAFKA-15241 – For tiered storage enabled clusters, compute tiered offset by keeping the respective epochs in scope. 

• KAFKA-15401 – Segment with corrupted index shouldn’t be uploaded to remote storage. 

• KAFKA-16073 – Fix to issue arising from the out-of-sync update of localLogStartOffset, leading to incorrect handling of consumer fetch requests and potential data access errors. 

• KAFKA-16511 – Under certain conditions, segments on remote storage may never get deleted. 

Improvements and fixes for issues identified with ZooKeeper to KRaft migration: 

• KAFKA-16003 – Fixes issue with not being able to successfully carry out a ZooKeeper to KRaft migration when in “dual-write” mode 

• KAFKA-16411 – Correctly migrates default client quota entities during ZooKeeper to KRaft migration. 

• KAFKA-16428 – Fixes bug where config changes notification znode may not get created during a ZooKeeper to KRaft migration. 

• KAFKA-16539 – Fixes an issue with not being able to update specific broker configurations in pre-migration mode, for ZooKeeper to KRaft migrations. 

• KAFKA-16563 – Fixes an issue which in certain situations causes the ZooKeper to KRaft migration can to hang. 

Some deprecations that were announced with this release: 

• KIP-896 – Client APIs released prior to Kafka 2.1 are deprecated and support for them will be removed with Kafka 4.0. 

• KIP-970 – Deprecates a redundant endpoint to expose task configurations. It will be removed in 4.0. 

• KIP-1013 – Java 11 support deprecated and planned to be removed with Kafka 4.0. 

Some of the ones we’ll be working on making available soon: 

• KIP-963 – Add more metrics specifically to help with monitoring Tiered Storage. 

• KIP-978 – Allows being able to dynamically reload certificates with different Domain Names (DN) / Subject Alternative Names (SAN), making it easier to use short-lived certificates (even if their DN or SAN change) causing less disruptions and making operations faster. 

• KIP-938 – Support for more metrics to measure KRaft performance. 

We are continuing to monitor the evolution of Tiered Storage and support for migration of ZooKeeper to KRaft with each release of Kafka. As of 3.7.1, we recommend the use of KRaft only on new Kafka clusters, i.e. not a migration from ZooKeeper to KRaft. As for Tiered Storage, we welcome you to try the recently released Preview version on our managed platform. With the upcoming release of Kafka 3.8, we will reevaluate being able to release Tiered Storage in GA. 

With this new release, we have reviewed and updated the lifecycle states for our older supported Kafka versions (as per our lifecycle policy). To ensure you get the full benefit of our support and SLAs, please reach out to us and upgrade to a GA version. More details on all Kafka versions supported on our managed platform are available here.  

We encourage you to try your preferred Kafka version in your non-production environments to confirm compatibility with your Kafka clients prior to deploying it in production. If you need any help spinning up a new Kafka cluster or upgrading existing managed Kafka clusters, please feel free to get in touch with us via our support website.