Striim 3.9.6 documentation

Google PubSub Writer

Writes to an existing topic in Google Cloud Pub/Sub.

Only async mode is supported. Consequently, events may be written out of order.

property

type

default value

notes

Batch Policy

Java.lang.String

EventCount:1000, Interval:1m, Size:1000000

Cached data is written to the target every time one of the specified values is exceeded. With the default value, data will be written once a minute or sooner if the buffer contains 1000 events or 1,000,000 bytes of data. When the application is stopped any remaining data in the buffer is discarded.

Due to google-cloud-java issue #4757, the maximum supported value for EventCount is 1000.

Project ID

Java.lang.String

the project to which the PubSub instance belongs

PubSub Config

Java.lang.String

RetryDelay: 5ms, MaxRetryDelay:600s, TotalTimeout:10s, InitialRpcTimeout:10s, MaxRpcTimeout:100s, RetryDelayMultiplier:2.0, NumThreads:10

configuration options to be passed to PubSub

Service Account Key

Java.lang.String

The path (from root or the Striim program directory) and file name to the .json credentials file downloaded from Google (see Service Accounts). This file must be copied to the same location on each Striim server that will run this adapter, or to a network location accessible by all servers.

If a value for this property is not specified, Striim will use the $GOOGLE_APPLICATION_CREDENTIALS environment variable.

The service account must have the PubSub Publisher or higher role for the topic (see Access Control).

Topic

Java.lang.String

the topic to publish to

This adapter has a choice of formatters. See Supported writer-formatter combinations for more information.

Sample application:

CREATE APPLICATION GooglePubSubWriterTest;

CREATE SOURCE PosSource USING FileReader (
  wildcard: 'PosDataPreview.csv',
  directory: 'Samples/PosApp/appData',
  positionByEOF:false )
PARSE USING DSVParser (
  header:Yes,
  trimquote:false )
OUTPUT TO PosSource_Stream;

CREATE CQ PosSource_Stream_CQ
INSERT INTO PosSource_TransformedStream
SELECT TO_STRING(data[1]) AS MerchantId,
  TO_DATE(data[4]) AS DateTime,
  TO_DOUBLE(data[7]) AS AuthAmount,
  TO_STRING(data[9]) AS Zip
FROM PosSource_Stream;

CREATE TARGET GooglePubSubTarget USING GooglePubSubWriter (
  ServiceAccountKey:'my-pubsub-cb179721c223.json',
  ProjectId:'my-pubsub',
  Topic:'mytopic'
)
FORMAT USING JSONFormatter ()
INPUT FROM PosSource_TransformedStream;

END APPLICATION GooglePubSubWriterTest;