Striim 3.9.6 documentation

S3 Reader

Reads from Amazon S3.

See Supported reader-parser combinations) for parsing options.

property

type

default value

notes

accesskeyid

java.lang.String

Specify an AWS access key ID (created on the AWS Security Credentials page) for a user with read permissions (ListBucket, GetObject) on the bucket.

When Striim is running in Amazon EC2 and there is an IAM role with those permissions associated with the VM, leave accesskeyid and secretaccesskey blank to use the IAM role.

blocksize

java.lang.Integer

64

amount of data in KB for each read operation

bucketname

java.lang.String

S3 bucket to read from

clientconfiguration

java.lang.String

Optionally, specify one or more of the following property-value pairs, separated by commas.

If you access S3 through a proxy server, specify it here using the syntax ProxyHost=<IP address>,ProxyPort=<port number>,ProxyUserName=<user name>,ProxyPassword=<password>. Omit the user name and password if not required by your proxy server.

Specify any of the following to override Amazon's defaults:

  • ConnectionTimeout=<timeout in milliseconds>: how long to wait to establish the HTTP connection, default is 50000

  • MaxErrorRetry=<number of retries>: the number of times to retry failed requests (for example, 5xx errors), default is 3

  • SocketErrorSizeHints=<size in bytes>: TCP buffer size, default is 2000000

See http://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html for more information about these settings.

compressiontype

java.lang.String

Set to gzip when the files to be read are in gzip format. Otherwise, leave blank.

foldername

java.lang.String

Specify a folder within the bucket, or leave blank to read from the root.

objectnameprefix

java.lang.String

The start of the names of the files to be read. For example, myfile will read myfile*.*. Specify * to read all files.

secretaccesskey

com.webaction. security.Password

Specify the AWS secret access key for the specified access key.

The output type is WAevent except when using Avro Parser  or JSONParser.

Example:

CREATE SOURCE S3Source USING S3Reader (
  bucketname:'MyBucket',
  objectnameprefix:'posdata',
  accesskeyid:'********************',
  secretaccesskey:'****************************************',
  foldername:'MyFolder'
)
PARSE USING DSVParser ()
OUTPUT TO S3SourceStream;