Configuring Kafka for persisted streams

Kafka streams may be persisted to Striim's internal Kafka server or to an external Kafka server.

Using Striim's internal Kafka server

Warning

On Windows, Zookeeper and Kafka do not shut down cleanly. (This is a well-known problem.) Before you restart Kafka, you must delete the files they leave in c:\tmp. Alternatively, look on stackoverflow.com for instructions on running Zookeeper and Kafka as services on Windows, or run an external Kafka server in a Linux virtual machine.

The default property set for the internal Kafka server that is installed with Striim at Striim/Kafka is Global.DefaultKafkaProperties:

jmx.broker=localhost:9998, 
bootstrap.brokers=localhost:9092, 
zk.address=localhost:2181

To change properties in an existing property set, see ALTER PROPERTYSET.

If you installed Striim with the JAR installer as discussed in Install Striim Platform for evaluation purposes and enabled Kafka in the setup wizard, it will start automatically. If you did not enable Kafka during installation, you may do so by re-running the setup wizard in the Strim/bin directory (WebConfig.exe for Windows, WebConfig for Mac, or WebConfig.sh for Linux).

If you installed Striim from a DEB, RPM, TGZ, or ZIP package as discussed in Running Striim in Ubuntu, Running Striim in CentOS, or Running Striim as a process, start Kafka as follows:

Open a terminal.
Change to Striim/Kafka, and enter bin/zookeeper-server-start.sh config/zookeeper.properties (this will start Zookeeper).
Open another terminal.
Change to Striim/Kafka and enter JMX_PORT=9998 bin/kafka-server-start.sh config/server.properties (this will start Kafka).

You can then persist Kafka streams using the default property set.

Using an external Kafka server

When using an external Kafka server, to handle Striim's maximum batch size the following entries in config/server.properties must have at least these minimum values:

message.max.bytes = 43264200
replica.fetch.max.bytes = 43264200
socket.request.max.bytes=104857600

To support persisting streams to an external server, use the Tungsten console to create a custom Striim property set using the following syntax:

CREATE PROPERTYSET <name> (
  bootstrap.brokers:'<bootstrap IP address>:<port>',
  jmx.broker:'<jmx IP address>:<port>'),
  zk.address:'<zookeeper IP address>:<port>',
  partitions:'<number of partitions to use>'
  kafkaversion:'{0.8|0.9|0.10|0.11|2.1}';

If not specified, partitions defaults to 200.

To change properties in an existing property set, see ALTER PROPERTYSET.

Using Kafka SASL (Kerberos) authentication with SSL encryption

To use SASL authentication with SSL encryption, do the following:

Get the files krb5.conf , principal.keytab , server.keystore.jks, and server.truststore.jks from your Kafka administrator and copy them to the Striim server's file system outside of the Striim program directory, for example, to /etc/striim/kafkaconf.

In the same directory, create the file jaas.conf, including the following lines, adjusting the keyTab path and principal to match your environment:

KafkaClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  storeKey=true
  doNotPrompt=true
  client=true
  keyTab="/etc/striim/kafkaconf/principal.keytab"
  principal="principal@example.com";
};

Add the following to Striim's Java environment:

JAVA_SYSTEM_PROPERTIES=" \
-Djava.security.krb5.conf='/etc/striim/kafkaconf/krb5.conf' \
-Djava.security.auth.login.config='/etc/striim/kafkaconf/kafka_server_jaas.conf' "

Include the following properties in your Kafka stream's property set or KafkaReader or KafkaWriter KafkaConfig, adjusting the paths to match your environment and using the passwords provided by your Kafka administrator. For KafkaConfig, replace the commas with semicolons.

security.protocol=SASL_SSL,
sasl.kerberos.service.name=kafka,
ssl.truststore.location=/etc/striim/kafkaconf/server.truststore.jks,
ssl.truststore.password=password,
ssl.keystore.location=/etc/striim/kafkaconf/server.keystore.jks,
ssl.keystore.password=password,
ssl.key.password=password

Using Kafka SASL (Kerberos) authentication without SSL encryption

To use SASL authentication without SSL encryption, do the following:

Get the files krb5.conf and principal.keytab from your Kafka administrator and copy them to the Striim server's file system outside of the Striim program directory, for example, to /etc/striim/kafkaconf.

In the same directory, create the file jaas.conf, including the following lines, adjusting the keyTab path and principal to match your environment:

KafkaClient {
  com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  storeKey=true
  doNotPrompt=true
  client=true
  keyTab="/etc/striim/kafkaconf/principal.keytab"
  principal="principal@example.com";
};

Add the following to Striim's Java environment:

JAVA_SYSTEM_PROPERTIES=" \
-Djava.security.krb5.conf='/etc/striim/kafkaconf/krb5.conf' \
-Djava.security.auth.login.config='/etc/striim/kafkaconf/kafka_server_jaas.conf' "

Include the following properties in your Kafka stream's property set or KafkaReader or KafkaWriter KafkaConfig . For KafkaConfig, replace the comma with a semicolon.
```
security.protocol=SASL_PLAINTEXT,
sasl.kerberos.service.name=kafka
```

Using Kafka SSL encryption without SASL (Kerberos) authentication

To use SSL encryption without SASL authentication, do the following:

Get the files server.truststore.jks and server.keystore.jks from your Kafka administrator and copy them to the Striim server's file system outside of the Striim program directory, for example, to /etc/striim/kafkaconf.
Include the following properties in your Kafka stream property set or KafkaReader or KafkaWriter KafkaConfig, adjusting the paths to match your environment and using the passwords provided by your Kafka administrator. For KafkaConfig, replace the commas with semicolons.
```
security.protocol=SSL,
ssl.truststore.location=/etc/striim/kafkaconf/server.truststore.jks,
ssl.truststore.password=password,
ssl.keystore.location=/etc/striim/kafkaconf/server.keystore.jks,
ssl.keystore.password=password,
ssl.key.password=password
```

Using Kafka without SASL (Kerberos) authentication or SSL encryption

To use neither SASL authentication nor SSL encryption, do not specify security.protocol in the KafkaReader or KafkaWriter KafkaConfig or in your Kafka stream's property set.

Additional properties for Kafka streams

Use these properties only in Kafka stream property sets, not with KafkaReader or KafkaWriter. Use single quotes around the values.

Kafka property	default value	notes
partitions	`'200'`	the maximum number of Kafka partitions to be used if the stream is partitioned; if the stream is not partitioned, only one partition is used and this value is ignored
replication.factor	`'1'`	the number of replicas to keep for each event; if this is greater than the number of brokers, creation of a topic will fail