Apache Kafka REST Proxy
Please note that KRaft is not compatible with the Kafka REST Proxy add-on, and the Karapace REST Proxy should be used instead.
The Kafka Rest Proxy is an addon which can be added when creating an Instaclustr Managed Apache Kafka Cluster. Note that enabling this add-on will add 20% to standard costs. If you would like to add the Kafka Rest Proxy to an existing cluster, you should contact support to have it added.
To create a new Kafka cluster with the Kafka REST Proxy add-on, make sure to enable Kafka REST Proxy under the Enterprise Add-Ons subsection on the Kafka setup page.
You can also choose to integrate the Kafka REST Proxy with a Kafka Schema Registry of your choice. To do so, simply click the checkbox and enter the server details for the Kafka Schema Registry. Alternatively, you can create a local Kafka Schema Registry on the cluster and choose to integrate the Kafka REST Proxy with the local one.
Configuring the Kafka Rest Proxy
To start using the Kafka Rest Proxy, you must first open the firewall to your client application IP address. Once your cluster is up and running, go to Firewall Rules and add your IP address to the Kafka REST Proxy Allowed Addresses.
Connecting to the Kafka Rest Proxy
The Kafka Rest Proxy is hosted in our Instaclustr hosted zone, cnodes.io. You can access the Kafka Rest Proxy by using the subdomain we have created.
Connection information can be retrieved on the Connection Info page for the cluster running the Kafka REST Proxy.
This page also includes the username and automatically generated password, plus code samples for testing connectivity to your Kafka REST Proxy servers.
Important information regarding Kafka Rest Proxy access
We currently do not offer the ability to restrict access via ACLs for Kafka Rest Proxy clients, the Kafka Rest Proxy service connects to your Apache Kafka cluster and has access to all topics stored in the cluster.
Please make sure you understand this restriction before adding the Kafka Rest Proxy service to your cluster.
Using the Kafka Rest Proxy
The Kafka Rest Proxy operates as a RESTful web API, any application that can send and receive HTTPS messages can be configured to operate with the application.
Special attention should be paid to the Content Type headers sent with each request. The same Content Type headers must be sent when setting up and configuring consumer instances and producing Kafka topics, otherwise, the requests may fail or return 404 NOT FOUND.
For more information about what content types are supported, refer to the following documentation:
https://docs.confluent.io/5.0.0/kafka-rest/docs/api.html
Example:
- The following examples will illustrate a few basic commands using the CURL command on a Unix based client machine.
- The consumer instance will be named “my_consumer_instance”
- The Content Type header used will be application/vnd.kafka.json.v2+json
- Consumer instances are removed after 5 minutes of idle time.
- Listing the topics in a Kafka cluster
|
1 2 3 |
Request curl -XGET -u restUser:restPassword https://kafka-proxy.149d286a4168457da1705e4f7016faa7.cnodes.io:8083/topics |
|
1 2 3 |
Response 200 OK ["instaclustr-sla"] |
- Create a consumer instance in the consumer group named ‘consumer1’
|
1 2 3 |
Request curl -XPOST -u restUser:restPassword https://kafka-proxy.149d286a4168457da1705e4f7016faa7.cnodes.io:8083/consumers/consumer1 --data '{"name": "my_consumer_instance", "format": "json", "auto.offset.reset": "earliest"}' --header "Content-Type: application/vnd.kafka.json.v2+json" |
|
1 2 3 |
Response 200 OK {"instance_id":"my_consumer_instance","base_uri":"https://34.239.232.248:8083/consumers/consumer1/instances/my_consumer_instance"} |
*Note: Although Instaclustr provides 3 Kafka Rest Proxy nodes, when creating a consumer instance you will be returned a URL for a specific node IP address.
This is expected and necessary behavior, as the consumer instances are stateful and each instance maintains the state independently.
Subsequent requests for that consumer instance must be made to the same IP address.
- Subscribe consumer instance to the topic “test”
|
1 2 |
Request curl -XPOST -u restUser:restPassword https://34.239.232.248:8083/consumers/consumer1/instances/my_consumer_instance/subscription --data '{"topics": ["test"]}' --header "Content-Type: application/vnd.kafka.json.v2+json" |
|
1 2 |
Response 204 NO CONTENT |
- Produce a message to the topic “test”
|
1 2 |
Request curl -XPOST -u restUser:restPassword https://kafka-proxy.149d286a4168457da1705e4f7016faa7.cnodes.io:8083/topics/test --data '{"records": [{"key": "hello","value": "world"}]}' --header "Content-Type: application/vnd.kafka.json.v2+json" |
Note: producing can be done to any Kafka Rest Proxy node
|
1 2 3 |
Response 200 OK {"offsets":[{"partition":2,"offset":0,"error_code":null,"error":null},{"partition":1,"offset":0,"error_code":null,"error":null},{"partition":2,"offset":1,"error_code":null,"error":null}],"key_schema_id":null,"value_schema_id":null} |
- Consume the produced message
|
1 2 |
Request curl -XGET -u restUser:restPassword https://34.239.232.248:8083/consumers/consumer1/instances/my_consumer_instance/records --header "Accept: application/vnd.kafka.json.v2+json" |
|
1 2 3 |
Response 200 OK [{"topic":"test","key":"hello","value":"world","partition":0,"offset":1}] |
Troubleshooting
The most common issue for new users is you receive a 404 NOT FOUND when attempting to consume from your consumer instance.
There can be a few reasons this may occur
Mismatched “Content-Type” headers
Check your Content-Type headers are consistent for each request made to create and configure your consumer instance, any mismatch on datatype or API version will cause requests to fail.
The consumer instance exists on another Kafka Rest Proxy server
Ensure you are communicating with the correct server. Instaclustr offers 3 Kafka Rest Proxy nodes to produce and consume from, however, a single consumer instance only exists on the server it was created.
To use this consumer instance, you must keep connecting to the same server by its IP address. To facilitate this pattern, in addition to the ID of the consumer instance, the Kafka Rest Proxy will also return the fully qualified URL your application should use.
Idle consumer instances are deleted after 5 minutes
The final reason could be the consumer instance has been deleted, either because it was idle for longer than 5 minutes or the Kafka Rest Proxy application was restarted.
The Kafka Rest Proxy application was restarted
While Instaclustr does everything it can to avoid restarting your applications, they may get restarted during general maintenance or troubleshooting operations.
To account for both scenarios, you can ensure your client code is robust by attempting to use the existing consumer instance, and if you receive a 404 response, creating a new consumer instance and using that one.
Further reading
The example above is a simple illustration of a basic produce/consume workflow.
Instaclustr supports provides and supports all the available features and operations provided by the Kafka Rest Proxy.
Please consult the API reference for a comprehensive guide to these features.
The Kafka Rest Proxy enables you to integrate an Instaclustr Managed Kafka cluster with any system that can communicate with a simple HTTPS RESTful API.
Support information
Due to the change in licensing agreement, Instaclustr has forked the original Kafka-Rest solution and will maintain updates at this GitHub repository.
By Instaclustr Support