Skip to main content

Testing KafkaWriter performance

Striim includes two utility scripts that may be useful in tuning your Kafka configuration. Striim/tools/enhanced-producer-perf.sh writes to Kafka directly from the Striim host. Striim/tools/standalone-kafkawriter-perf.sh writes with KafkaWriter.

Script arguments

The arguments are the same for both scripts:

argument

notes

file

The file to be used for testing.

Specify none to use an in-memory data generator, in which case you must also specify record-size and num-records. 

input-format

If using a file, set to avro or dsv, or omit or set to none to read the file as is.

topic

The Kafka topic to write to.

producer-props

A file containing the Kafka server properties. See striim/tools/props for an example.

output-format

Set to json to format the data or omit or set to none to write as is.

record-size

If using --file none, the size of each generated record in bytes.

num-records

If using --file none, the number of records to write.

mode

Set to sync for synchronous writes, or omit or set to async for asynchronous writes (see Setting KafkaWriter's mode property: sync versus async).

single-partition

Set to true to write to all partitions of the topic. Omit or set to false to write only to partition 0.

Script usage examples

For example, the following will generate one million 120-byte records and write them directly to Kafka.

./enhanced-producer-perf.sh --file none --topic test01  --producer-props props
 --record-size 120 --num-records 1000000 

Output from that command would be similar to:

Configuration used for testing Apache Kafka Producer:
File : none
Topic : test01
Partitions : 1
Single Partition : false
Mode : async
Output format : none
Num Records : 1000000
Record Size : 120
---------------------------------
Final Stats
Records sent : 1000000
Records/sec : 521920.6680584551
MB/sec : 62.63048016701461
Avg Latency(ms) : 559.206501
Max Latency(ms) : 1054
Avg message size (bytes): 120.0
Number of times Producer send called : 1000000
Avg time between 2 Producer send calls(ms) : 0.003199727455364971
Max time between 2 Producer send calls(ms) : 210
---------------------------------

The following will generate the same number of records of the same size but write them using KafkaWriter:

./standalone-kafkawriter-perf.sh --file none --topic test02  --producer-props props
 --record-size 120 --num-records 1000000

Output would be similar to:

Configuration used for testing Striim Kafka Writer:
File : none
Topic : test02
Partitions : 1
Single Partition : false
Mode : async
Output format : none
Num Records : 1000000
Record Size : 120
---------------------------------
Final Stats
Records sent : 1000000
Records/sec : 84918.47826086957
MB/sec : 3.9817331861413043
Avg Latency(ms) : 0.912115
Max Latency(ms) : 10018
Avg message size (bytes): 46.0
Number of times Producer send called : 1000000
Avg time between 2 Producer send calls(ms) : 0.011775
Max time between 2 Producer send calls(ms) : 10011