Striim 3.9.6 documentation

File Reader

Reads files from disk using a compatible parser.

You can create FileReader sources in the web UI using Source Preview.

See Supported reader-parser combinations) for parsing options.

When used with DSV Parser, the type for the output stream can be created automatically from the file header (see Creating the FileReader output stream type automatically).

Striim also provides templates for creating applications that read from files and write to various targets. See Creating a new application using a template for details.

property

type

default value

notes

blocksize

java.lang.Integer

64

amount of data in KB for each read operation

compressiontype

java.lang.String

Set to gzip when wildcard specifies a file or files in gzip format. Otherwise, leave blank.

directory

java.lang.String

Specify the path to the directory containing the file(s). The path may be relative to the Striim installation directory (for example, Samples/PosApp/appdata) or from the root.

includesubdirectories

java.lang.Boolean

False

Set to True if the files are written to subdirectories of the Directory path, for example, if each day's files are in a subdirectory named by date.

positionbyeof

java.lang.Boolean

True

  • If set to True, reading starts at the end of the file, so only new data is acquired.

  • If set to False, reading starts at the the beginning of the file and then continues with new data.

  • When FileReader is used with a cache, this setting is ignored and reading always begins from the beginning of the file.

  • When you create a a FileReader using Source Preview, this is set to False.

rolloverstyle

java.lang.String

Default

Set to log4j if reading Log4J files created using RollingFileAppender.

skipbom

java.lang.Boolean

True

If set to True, when the wildcard value specifies multiple files, Striim will read the Byte Order Mark (BOM) in the first file and skip the BOM in all other files. If set to False, it will read the BOM in every file.

wildcard

java.lang.String

name of the file, or a wildcard pattern to match multiple files (for example, *.xml)

The output type is WAevent except when using Avro Parser  or JSONParser.

An example from the PosApp sample application:

CREATE SOURCE CsvDataSource USING FileReader (
  directory:'Samples/PosApp/appData',
  wildcard:'posdata.csv',
  positionByEOF:false
)
PARSE USING DSVParser (
  header:Yes,
  trimquote:false
)
OUTPUT TO CsvStream;

See PosApp for a detailed explanation and MultiLogApp for additional examples.

Creating the output stream type automatically

When FileReader is used with DSV Parser, the type for the output stream can be created automatically from the file header using OUTPUT TO <stream name> MAP(filename:'<source file name>') . For example:

CREATE SOURCE CsvDataSource USING FileReader (
  directory:'Samples/PosApp/appData',
  wildcard:'posdata*.csv',
  positionByEOF:false
)
PARSE USING DSVParser (
  header:Yes,
  trimquote:false
)
OUTPUT TO CsvStream MAP(filename:’posdata*.csv’);

Notes:

  • The specified source file must exist when the source is created.

  • The header must be the first line of the file (the HeaderLineNo setting is ignored by MAP).

  • If multiple files are specified by the wildcard property, the header will be taken from the first one read.

  • All files must be like the first one read, with headers in the first line and the same number of fields.