Instaclustr is pleased to announce the general availability of Apache Kafka and Kafka Connect version 3.0.0 on the Instaclustr Managed Platform. The new version is available paired with version 3.0.0 of Kafka Connect.

Apache Kafka is the leading distributed event streaming platform and is open source under the Apache 2.0 License. Instaclustr provides our customers with managed and supported open source software allowing them to avoid the risks of both vendor lock-in and unsupported software with open source that’s really open source.

Compared to the previous release, version 3.0.0 of Apache Kafka includes some new features, deprecations and removal of previously deprecated items, improvements to KRaft (Apache Kafka’s inbuilt component meant to obviate the need for Apache ZooKeeper in the future). It’s also important to be aware that our preview of Kafka 3.0.0 is still leveraging Apache ZooKeeper, in line with the project’s recommendations, as KRaft is not yet suitable for production workloads. However, we will be making available a preview of ZooKeeper-less Kafka in early 2022.

Following are some highlights from the changes contained in Kafka 3.0.0:

Apache Kafka

• Apache ZooKeeper upgraded to version 3.6.3:

This version upgrade follows on from ZooKeeper version 3.5.9, which was included with Apache Kafka 2.7.1 and later. This release contains many bug fixes (including CVE fixes) and some new features. For more details, please refer to the Apache ZooKeeper 3.6.3 Release Notes.

• Producer will enable strongest delivery guarantee by default (KIP-679):

This change modifies certain defaults (enable.idempotence is set to true; acks is set to all) in the Producer config, along with some other modifications, to make record delivery guarantees stronger by default. As a modification to the default for enable.idempotence could cause compatibility issues for some users, it is advised you go through the migration plan outlined in KIP-679.

• Increase default consumer session timeout (KIP-735):

As part of this change, the default consumer session timeout has been increased from 10s to 45s to reduce the risk of spurious failures causing unnecessary rebalances to be triggered. Spurious failures can occur, for example, due to transient network/load failures. Such situations can lead to two rebalances being triggered: once when a group member failure is detected and later when that member rejoins the group. With the new session timeout, the member should have sufficient time to reconnect with the coordinator.

•  Change Kafka streams default replication factor config (KIP-733):

The default replication.factor=1 puts the onus on users to change the config when the application moves to production (as best practice recommends RF=3). For Kafka versions higher than 2.4.0, users can look to change the config to -1 ( to pick the broker default for replication factor), however, starting streams 3.0.0 this is no longer needed as the streams replication factor config will now be -1 by default.

• Some performance related improvements:

• • Update FindCoordinator to resolve multiple Coordinators at a time (KIP-699): Currently, to look up the coordinator for a resource, the FindCoordinator API can be used. To do this when multiple consumer groups are involved, the FindCoordinator would need to be called once for each consumer group, which can negatively affect performance. With KIP-699, FindCoordinator has been renamed to FindCoordinators, and now allows multiple such requests to be clubbed into one call, with the result containing a corresponding number of coordinators.

• • Extend OffsetFetch requests to accept multiple consumer group IDs (KIP-709): This change extends the functionality of the OffsetFetch API to be able to fetch offsets for multiple consumer groups. This was previously not the case as OffsetFetch could only support fetching the offset for one consumer group at a time.

• Significant deprecations:
  • MirrorMaker 1 is deprecated; will be removed from Kafka 4.0 (KIP-720)
  • Message formats v0 and v1 are deprecated; support will drop in Kafka 4.0 (KIP-724)
  • Java 8 is deprecated; support will drop in Kafka 4.0 (KIP-750)
  • Scala 2.12 is deprecated; support will drop in Kafka 4.0 (KIP-751)
    
    
• API deprecations that were removed:
  • Remove deprecated —zookeeper in topicCommands (KAFKA-12596)
  • Remove deprecated —zookeeper in ReassignPartitionsCommands (KAFKA-12597)
  • Remove deprecated —zookeeper in ConfigCommand (KAFKA-12598)
  • Remove deprecated kafka-preferred-replica-election command (KAFKA-8405)
  • Remove deprecated SimpleAclAuthorizer (KAFKA-12590)

Apache Kafka Connect

• Connector API to restart connector and tasks (KIP-745):

Currently, if a Connector or Task instance started by Kafka Connect experiences an error, it is marked as failed and exposed as such through the Connect REST API. To restart them, another REST API must be called once for each such instance. This change enables a single call to the Connect REST API to restart all failed instances.

• Removal of Connect’s internal converter properties (KIP-738):

The internal.key.converter and internal.value.converter Connect worker properties, and other properties prefixed with those names were deprecated with Kafka 2.0.0. Now, with Kafka 3.0.0, they’ve been removed and instead JsonConverter will be used with both key and value converters. Please follow the migration plan provided on the KIP-738 page if you’re impacted by this change.

If you would like to try our Managed Apache Kafka or Apache Kafka Connect service, you can sign up for a 30-day free trial.