-
- Technical
Apache Kafka® “Kongo” 6.1: Production Kongo on Instaclustr
In this blog we deploy the Kongo IoT application to a production Kafka cluster, using Instraclustr’s Managed Apache Kafka service on AWS. We explore Kafka cluster creation and how to deploy the Kongo code. Then we revisit the design choices made previously regarding how to best handle the high consumer fan out of the Kongo...
Learn MorePaul Brebner
June 29, 2018 -
- Technical
Apache Kafka® “Kongo” 5.3: Kongo Streams Example
Introduction In the previous blog we tried a simple Kafka Streams application for Cluedo. It relied on a KTable to count the number of people in each room. In this blog, we’ll extend this idea and develop a more complex streams application to keep track of the weight of goods in trucks for our Kongo...
Learn MorePaul Brebner
June 20, 2018 -
- Technical
Can We Solve a Mystery with Apache Kafka® Streams?!–Kongo 5.2
In this blog, we’ll look at some simple Apache Kafka Streams examples using the murder mystery game Cluedo as a simple problem domain. Dr Black has been murdered in the Billiard Room with a Candlestick! Whodunnit?! There are six suspects and a mansion with multiple rooms. The suspects are: Miss Scarlet Professor Plum Mrs Peacock...
Learn MorePaul Brebner
May 29, 2018 -
- Technical
Kongo 5.1: Apache Kafka® Streams Introduction
Abstract Apache Kafka Streams is a framework for stream data processing. In this blog, we’ll introduce Kafka Streams concepts and take a look at one of the DSL operations, Joins, in more detail. In the next blog, we’ll have a look at some more complete Kafka Streams examples based on the murder mystery game Cluedo....
Learn MorePaul Brebner
May 29, 2018 -
- Technical
Apache Kafka “Kongo” Part 4.2: Connecting Kafka to Cassandra with Kafka Connect
Here’s the Kongo code and sample connect property files for this blog. Kafka Connect is an API and ecosystem of 3rd party connectors that enables Kafka to be easily integrated with other heterogeneous systems without having to write any extra code. This blog focuses on a use case extending the Kongo IoT application to stream...
Learn MorePaul Brebner
May 25, 2018 -
- Technical
Apache Kafka “Kongo” Part 4.1: Connecting Kafka to Cassandra with Kafka Connect
Kafka Connect is an API and ecosystem of 3rd party connectors that enables Kafka to be easily integrated with other heterogeneous systems without having to write any extra code. This blog focuses on a use case extending the Kongo IoT application to stream events from Kafka to Apache Cassandra using a Kafka Connect Cassandra Sink....
Learn MorePaul Brebner
May 09, 2018 -
- Technical
Apache Kafka® Connect Architecture Overview
Kafka Connect is an API and ecosystem of 3rd party connectors that enables Apache Kafka to be scalable, reliable, and easily integrated with other heterogeneous systems (such as Cassandra, Spark, and Elassandra) without having to write any extra code. This blog is an overview of Kafka Connect Architecture with a focus on the main Kafka...
Learn MorePaul Brebner
May 09, 2018 -
- Technical
“Kongo” Part 3: Apache Kafka®: Kafkafying Kongo—Serialization, One or Many topics, Event Order Matters
Kafkafying: the transformation of a primitive monolithic program into a sophisticated scalable low-latency distributed streaming application (c.f. “An epidemic of a zombifying virus ravaged the country”) Steps for Kafkafying Kongo In the previous blog (“Kongo” Part 2: Exploring Apache Kafka application architecture: Event Types and Loose Coupling) we made a few changes to the original...
Learn MorePaul Brebner
April 26, 2018