NetApp is excited to announce the availability In Public Preview of Tiered Storage for Instaclustr for Apache Kafka®.  

Tiered Storage is a game-changing feature that allows Kafka users to offload their historical data to less expensive, scalable storage systems without sacrificing the ability to query and utilize this data in real-time streaming applications.  

With this new capability, Instaclustr customers can now significantly reduce the costs associated with data retention, while maintaining the performance and reliability they have come to expect from the Instaclustr Managed Platform. 

Some key benefits of Tiered Storage with Kafka: 

• Storage and Replication Cost Reduction: You can reduce your overall storage costs while retaining your data. This makes it accessible for future analytics and processing by moving infrequently accessed data to more cost-effective storage. Additionally, data that would have previously been replicated across multiple brokers for fault tolerance is instead moved to remote storage. This leads to a reduction in storage and network (during cluster topology changes) costs compared to data on the local storage.
• (Theoretically) Indefinite Retention: You can now keep data indefinitely without worrying about the physical limitations of the local storage attached to your Kafka cluster by having data moved to limitless remote storage.   
• Seamless Integration and Data Access: The Tiered Storage feature integrates smoothly with existing Kafka clusters managed by Instaclustr, ensuring a straightforward setup process with simple one-click enablement. For Kafka clients, access to both the hot data on the brokers and the cold data in the tiered storage is seamless and requires no changes to existing clients. Tiered Storage is smart enough to identify which data is needed and make it available, when requested, to the Kafka clients.
• Decoupling of Storage and Compute: By separating the storage layer from the Kafka brokers, Tiered Storage allows for independent scaling of storage and compute resources. For example, as data volumes grow, the storage can be expanded more economically by adding more capacity in the tiered storage layer rather than scaling up the broker storage or adding more brokers to the cluster.  
• High Performance: Even with data offloaded to secondary storage, Tiered Storage maintains high throughput and low-latency performance, with negligible impact on access latencies (for common workloads), especially in scenarios where a broker needs to be rebuilt, or where out of sync consumers need to catchup. 

Tiered Storage is offered as an Enterprise Add-On but is included in Instaclustr Enterprise Pricing. Once Enterprise Pricing is applied to your cluster, all enterprise features such as PCI or PrivateLink are available to you at no extra cost. Please contact your Customer Success Representative for more details around pricing applicable to you. 

For our Managed Platform customers creating new clusters with Kafka 3.6.1 and greater, they will have the option to enable Tiered Storage during the setup process. For customers who already have existing Kafka clusters running version 3.6.1 or greater, please open a ticket with our Support team providing details of the cluster you would like this feature enabled on.

Information to help you understand how Tiered Storage in Kafka works is available here, while steps on how to enable it for your Instaclustr for Apache Kafka cluster are provided here. 

When using features in Public Preview, please bear in mind that they are not covered by our standard SLAs and are not recommended for production purposes. In the case of Tiered Storage, it is still undergoing further development by the Kafka project to remove known limitations in future releases and is not marked as production-ready at the time of writing this article.