Introduction

Earlier this year Instaclustr announced the introduction of support for Karapace—an Apache 2.0 licensed, purely open source alternative to Confluent Schema Registry and REST Proxy. Karapace comes with all the accompanying benefits and most importantly no vendor lock-in.

This blog provides a comparison of Karapace Schema Registry and REST Proxy 3.2.0, and Schema Registry 5.0.4 and REST Proxy 5.0.0 offerings that Instaclustr provides as add-ons to our managed Apache Kafka^® service. 

Please note that all terminology used in this blog is in the context of Apache Kafka, unless otherwise specified. 

What Is Schema Registry?

Schema Registry is a Kafka add-on which provides a RESTful interface allowing storage, update, and version control of formatted schemas. Supported formats may include Avro, JSON, and Protobuf. 

A schema is a definition of the structure of the data format. Producers can register a schema via the Schema Registry interface and send data in registered formats. Schema Registry will send the formatted data (serialized) to the brokers for streaming. Before being consumed by consumers, data is sent from Kafka brokers to the Schema Registry interface again to retrieve the structure (deserialize) so that consumers can get the exact same structure used by the producer, where its structure, type, and meaning are clearly defined. 

Schema Registry provides a pathway for fast and continuous integration of potentially rapidly changing schemas over the life-cycle of a data pipeline. Additionally, Schema Registry stores version information for all schemas, allowing customers to have better visibility to be able to ensure compatibility between versions.

What Is the REST Proxy?

REST Proxy is an add-on that provides a RESTful interface directly to the Kafka clusters without using the native Kafka protocol or clients. By being light-weight and independent while also providing the freedom of building applications that access Kafka using languages not currently supported by Kafka clients, it expands the types of applications that can use Kafka. REST proxies for Kafka usually support actions including topic creation, deletion, as well as producing to or consuming from a topic.

Without REST Proxy one would need to use either the Kafka CLI tools or third-party language-specific Kafka clients, for example, kafka-python, to communicate and interact with a Kafka cluster. Alternatively, customers of Instaclustr for Apache Kafka could use the Instaclustr API tool to perform administrative tasks such as topic creation and deletion.

Karapace Schema Registry and REST Proxy

Karapace is an open source project, available under the Apache 2.0 license, which provides a schema registry as well as a REST Proxy component. Karapace Schema Registry is currently fully compatible with the Confluent Schema Registry up to version 6.1.1 on the API level. 

Initiated by Aiven and maintained by the open source community, the goal for the Karapace offering is to continue to add new features and fixes in a reasonable timeframe. Instaclustr has been an active contributor to the Karapace project having recently contributed implementations for the ProtoBuf format and support for schema references.

Confluent Schema Registry and REST Proxy

Instaclustr supports Confluent Schema Registry 5.0.4 and Confluent REST Proxy 5.0.0 add-ons for our managed Apache Kafka clusters which are the last versions of Confluent Schema Registry and Confluent REST Proxy available under the Apache 2.0 license.  Later versions of Schema Registry and REST Proxy offerings are available under the more restrictive Confluent Community License.

Instaclustr Offerings

Instaclustr managed Kafka clusters currently support both Karapace Schema Registry and REST Proxy, as well as Confluent Schema Registry and Rest Proxy. When creating a new Kafka cluster, customers of our Instaclustr for Apache Kafka managed service can select which pair of add-ons to enable.

Instaclustr started offering Karapace Schema Registry and REST Proxy in May 2022, as a managed add-on for our managed Kafka clusters, while also contributing to the Karapace open source project to evolve our offering in stability, scope, and functionality. Currently, on our managed platform we support Karapace Schema Registry and REST Proxy 3.2.0 as add-ons. As newer versions with more functionality suitable for our customers become available, we will test and make them available.

As for Confluent Schema Registry and Confluent REST Proxy, we currently offer versions 5.0.4 and 5.0.0, respectively. As mentioned above, due to licensing changes for Confluent’s tools, we are unable to provide customers with newer versions.

Our recommendation is that customers use Karapace. This is in keeping with our commitment of providing our customers with the latest open source options. Additionally, we intend to cease offering the Schema Registry and Rest Proxy add-ons for new clusters in the near future with support for existing users continuing for a transition period after that. We will provide advanced notification to customers of deprecation and end of life times as they are finalized.

Comparison

In the next sections will be comparisons of features between both options as currently supported by the Instaclustr for Apache Kafka managed service. Some features are only limited to specific offerings—for example, REST Proxy support for PrivateLink clusters is limited to the Karapace offering. 

Table: Overview of comparisons between Instaclustr supported Confluent and Karapace Schema Registry and REST Proxy.

Schema Formats

Different formats of schema provide diversity and multiple options, which can be convenient for users with different Kafka use cases and language choices. Apache Kafka currently supports schema formats including Avro, JSON, and Protobuf. Confluent Schema Registry 5.0.4 only supports schemas in Avro format, while the Karapace Schema Registry 3.2.0 supports all popular formats, Avro, JSON, and Protobuf. 

Schema Monitoring and Metrics

Monitoring for the Confluent Schema Registry supports a variety of metrics at both a global and an API endpoint level. Additional custom stat metrics can also be configured and plugged in. Global metrics report general health status for Kafka, such as activity of brokers and connection acceptance rate. These metrics provide a high-level overview of the Kafka cluster’s performance. 

At an API endpoint level, users could use these metrics to learn about per-endpoint status, such as frequency and latency of visit for specific services and sizes for requests. These metrics provide a detailed breakdown of the global view, leading to a better understanding of the Kafka cluster’s performance. 

The Karapace project has a roadmap item to work on supporting metrics for its schema registry. Instaclustr is aiming to help expedite the roll-out of this feature by contributing resources and implementation.

Authentication System

Confluent Kafka Schema Registry and REST Proxy provide TLS authentication for communications between client and the APIs as an advanced paid feature. Karapace Schema Registry and REST Proxy do not support TLS authentication.

However, Instaclustr provided add-ons for Confluent Schema Registry and REST Proxy, as well as Karapace Schema Registry and REST Proxy are all able to provide authentication between client and API endpoints by making use of Nginx under the hood.

PrivateLink support (for AWS)

Instaclustr provides PrivateLink functionality with full support for clusters of our Instaclustr for Apache Kafka managed service running on AWS. Add-ons for Karapace Schema Registry and Karapace REST Proxy are also supported. However, while Confluent Schema Registry is supported as well, Confluent REST Proxy is not due to technical constraints.

Conclusion 

In conclusion, while Instaclustr provides support for Confluent Schema Registry and REST Proxy, as well as Karapace Schema Registry and REST Proxy as add-ons, given the differences in licencing and our firm commitment to supporting open source projects—including Karapace—we believe that now, and in the future, Karapace will better serve the features and preferences of our customers.

Please get in contact with our Customer Support team if you  have any questions, or would like help with transitioning to Karapace.