Striim 3.9.6 documentation

Kinesis Writer

Writes to an Amazon Kinesis stream.

property

type

default value

notes

accesskeyid

java.lang.String

Specify an AWS access key ID (created on the AWS Security Credentials page) for a user with write permission on the stream.

When Striim is running in Amazon EC2 and there is an IAM role with that permission associated with the VM, leave accesskeyid and secretaccesskey blank to use the IAM role.

batchpolicy

java.lang.String

Size:900000, Interval:1

The batch policy includes eventcount and interval (see Setting output names and rollover / upload policies for syntax). Events are buffered locally on the Striim server and sent as a batch to the target every time either of the specified values is exceeded. When the app is stopped, any remaining data in the buffer is discarded. To disable batching, set to EventCount:1,Interval:0.

With the default setting, data will be written every second or sooner if the an event pushes the buffer past 900,000 bytes. The buffer will expand as necessary to include that last event in the batch.

mode

java.lang.String

Sync

Do not change from default.

partitionkey

java.lang.String

Optionally, specify a field to be used to partition the events among multiple shards. See Kafka Writer for more details.

regionname

java.lang.String

the AWS region of the stream (see AWS Regions and Endpoints)

secretaccesskey

java.lang.String

the secret access key for the stream

sessiontoken

java.lang.String

If you are using a session token (see GetSessionToken), specify it here. See also Temporary Security Credentials.

streamname

java.lang.String

the existing Kinesis stream to write to (will not be created if it does not exist)

This adapter has a choice of formatters. See Supported writer-formatter combinations for more information.

Limitations:

  • Only sync mode is supported.

  • Parallel threads are not supported.

  • Do not write to a target stream with more than 249 shards.

  • Do not merge, split, start or stop encryption on, or change the shard count of the target stream.

  • QUIESCE is not supported in this release.

  • If multiple writers write to the same shard, there may be duplicate events after Recovering applications (in other words, exactly-once processing cannot be guaranteed).

  • The maximum record length cannot exceed 1MB, including the metadata Striim adds to support recovery. (This is a limitation in Kinesis, not Striim.)

Example:

CREATE TARGET KinesisTest USING KinesisWriter (
  regionName:'us-east-1',
  streamName:'myStream',
  accesskeyid:'********************',
  secretaccesskey:'****************************************',
  partitionKey: 'merchantId'
)
FORMAT USING JSONFormatter ()
INPUT FROM PosSource_TransformedStream;