Overview: Cassandra Stress

This is the first in a series of blog posts I’m planning to create as part of my prep for my Cassandra summit talk ‘Load Testing Cassandra Applications’.

Cassandra-stress is a great utility for stress testing Cassandra. However, available documentation is a little sparse and it is not always entirely clear what load Cassandra-stress will generate in a given situation. In this series of blog posts, I plan to walk through a number of Cassandra stress scenarios examining exactly how Cassandra-stress behaves.

For this series, I will be using the latest 3.x version of Cassandra-stress. If I notice any differences related to a particular point version of Cassandra I will call them out.

In this first post, I will look at what is about the most basic Cassandra-stress command you can run:
cassandra-stress write n=10 -node x.x.x.x

I chose this command for two reasons: firstly, using a simple command will allow us to look at some of the basic functions that apply across any Cassandra-stress command and secondly, in almost all scenarios, you will want to execute a write to populate a cluster with data before running a read or mixed scenario.

Cassandra-stress

Let’s start by looking at the components of the command itself:

• cassandra-stress: invokes a shell script which in turn invokes the main function of the Java class org.apache.cassandra.stress.Stress
• write: execute write operations (other options being read, mixed, user, counter_write and counter_read)
• n=10: execute 10 operations
• -node x.x.x.x: the address of a node in the cluster to establish the initial connection

Filling in with defaults, here are all the settings Cassandra-stress will actually use for the run (from my in-progress implementation of CASSANDRA-11914):

Step by Step

As you can see, there is a lot going on behind the scenes. So. let’s walk-through step by step what Cassandra-stress actually does when you execute this command:

1. The options provided through the command line are parsed and filled in with a whole range of default options as necessary (will touch on important defaults below and more in future articles). Interestingly, at this stage a valid cassandra.yaml will need to be loaded. However, as far as I could tell this is just a side effect of cassandra-stress using some core Cassandra classes and the contents of the cassandra.yaml have no effect on the actual cassandra-stress operations.
2. The Cassandra Java driver is used to connect to the node specified in the command line. From this node, the driver retrieves the node list and token map for the cluster and then initiates a connection to each node in the cluster.
3. A create keyspace (if one doesn’t already exist) command is executed with the following definition:
   CREATE KEYSPACE IF NOT EXISTS keyspace1 WITH durable_writes = true AND replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };

   1 2 3 4 5 6

   CREATE KEYSPACE IF NOT EXISTS keyspace1 WITH durable_writes = true AND replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };

   While this definition is a reasonable choice for a simple default, it’s important to note that this is unlikely to be representative of a keyspace you would want to run in production. By far the most common production scenario would be to use NetworkTopologyStrategy and a replication factor of 3. To have cassandra-stress create a keyspace with this strategy you would need to drop any existing keyspace and add the following parameters to the cassandra-stress command line:
   -schema replication(strategy=NetworkTopologyStrategy,DC_NAME=3)
   Replace DC_NAME with the actual name of your Cassandra data center. On some systems you may also need to escape the brackets ie. replication(...)

4. Once the keyspace is created, cassandra-stress creates two tables in the keyspace: standard1 and counter1. We’ll ignore counter1 for now as it’s not used in this test. The definition of the standard1 table created is as follows:
   CREATE TABLE IF NOT EXISTS standard1 (key blob PRIMARY KEY, "C0" blob, "C1" blob, "C2" blob, "C3" blob, "C4" blob) WITH COMPACT STORAGE AND compression = {};

   1 2 3

   CREATE TABLE IF NOT EXISTS standard1   (key blob PRIMARY KEY,    "C0" blob, "C1" blob, "C2" blob, "C3" blob, "C4" blob) WITH COMPACT STORAGE AND compression = {};

   While, again, this is a reasonable choice for a simple default, there are a few characteristics to keep in mind if you are trying to draw conclusions from performance using this table definition:

   • There is no clustering key, so 1 row per partition – potentially very different performance to a scenario with many rows per partition.
   • Compression is disabled – the overhead of compression is typically not huge but could be significant.
   • Compact Storage is enabled – this is not enabled by default and will result in smaller representation of the data on disk (although minimal difference with Cassandra 3.x).

   I’ll cover options for using different schemas in a later installment of this series.

5. cassandra-stress will attempt to make a jmx connection to the nodes in the cluster to collect garbage collection stats. If the attempt fails, the run will proceed without collection garbage collection stats.
6. Next, cassandra-stress will run a warmup. This is a standard practice in load testing to reduce variation from start-up variation such as code being loaded to memory and JVM hotspot compilers. The number of warm-up iterations is the lesser 25% of the target of operations or 50k – 2 operations in this trivial example. The warm-up operations are basically the same as the test operations except not timed so I won’t go into them in detail.
7. We’ve entered the actual load test phase. cassandra-stress creates 200 client threads and begins executing the target number of operations. In a real test, using the -rate option to control the number of client threads is a good way to control load on the cluster.
   The first attempted operation will create a CQL prepared statement as follows:
   UPDATE "standard1" SET "C0" = ?,"C1" = ?,"C2" = ?,"C3" = ?,"C4" = ? WHERE KEY=?
   Although we were probably expecting an INSERT statement, updates and inserts are identical in terms of Cassandra implementation so we can expect performance to be the same.This prepared statement will then be will be executed 10 times with different, random data generated for each execution. The statement will be executed with consistent level LOCAL_ONE.cassandra-stress seeds the random generation with a static string plus the column name and a seed number which for the write command defaults to sequentially used numbers from 1 to the number of operations. That means that each column will get different values but the set of values generated will be the same over multiples runs. Generating a static set of values is necessary for read tests but does have the side effect that if you were to run our sample operation (write n=10) 1000 times the end result would still be just 10 rows of data in the table.
8. Finally, cassandra-stress prints its results. Here’s an example from a run of this command:
   Results: Op rate : 545 op/s [WRITE: 545 op/s] Partition rate : 545 pk/s [WRITE: 545 pk/s] Row rate : 545 row/s [WRITE: 545 row/s] Latency mean : 35.9 ms [WRITE: 35.9 ms] Latency median : 37.2 ms [WRITE: 37.2 ms] Latency 95th percentile : 42.8 ms [WRITE: 42.8 ms] Latency 99th percentile : 42.8 ms [WRITE: 42.8 ms] Latency 99.9th percentile : 42.8 ms [WRITE: 42.8 ms] Latency max : 42.8 ms [WRITE: 42.8 ms] Total partitions : 10 [WRITE: 10] Total errors : 0 [WRITE: 0] Total GC count : 0 Total GC memory : 0.000 KiB Total GC time : 0.0 seconds Avg GC time : NaN ms StdDev GC time : 0.0 ms Total operation time : 00:00:00

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

   Results: Op rate                   :      545 op/s  [WRITE: 545 op/s] Partition rate            :      545 pk/s  [WRITE: 545 pk/s] Row rate                  :      545 row/s [WRITE: 545 row/s] Latency mean              :   35.9 ms [WRITE: 35.9 ms] Latency median            :   37.2 ms [WRITE: 37.2 ms] Latency 95th percentile   :   42.8 ms [WRITE: 42.8 ms] Latency 99th percentile   :   42.8 ms [WRITE: 42.8 ms] Latency 99.9th percentile :   42.8 ms [WRITE: 42.8 ms] Latency max               :   42.8 ms [WRITE: 42.8 ms] Total partitions          :         10 [WRITE: 10] Total errors              :          0 [WRITE: 0] Total GC count            : 0 Total GC memory           : 0.000 KiB Total GC time             :    0.0 seconds Avg GC time               :    NaN ms StdDev GC time            :    0.0 ms Total operation time      : 00:00:00

   Many of these results are self explanatory but some bear further explanation:
   Op rate is the rate of execution commands. Partition rate is that rate that partitions were visited (updated or read) by those commands and Row rate is that rate that rows were visited. For simple, single-row commands all three rates will be equal. The rates will vary in more complex scenarios where a single operation might visit multiple partitions and rows.
   Similarly, Total partitions is the total number of partitions visited during the test. It’s worth noting that this is not unique partitions so even in some write-only scenarios it may not reflect the total number of partitions created by the test.
   The GC statistics report on garbage collection and are zero in this case as JMX ports were blocked to the test cluster.

Conclusion

Well, that’s a lot to write about a simple test that inserts 10 rows into a table. Putting it together has helped improve my understanding of Cassandra-stress, I hope it’s useful for you too. In future installments, I’ll look some more into the different data generation operations, mixed test, and customers schemas using the YAML configuration file. Let me know in the comments if there are any particular areas of interest for future articles.

Click here for Part Two: Mixed Command
Click here for Part Three: Using YAML Profiles