Striim 3.9.7 documentation

Using the META() function

You can use the META() function to query the metadata map of the WAEvent type, which is used by the output streams of several Striim readers. For example, the following creates a stream containing invalid records:

CREATE STREAM ExceptionStream of ExceptionRecord;
CREATE CQ CQExceptionRecord
INSERT INTO ExceptionStream
SELECT data[0]
FROM CsvStream
WHERE META(CsvStream, ‘RecordStatus’).toString == ‘INVALID_RECORD’;

The elements of the metadata map vary depending on the reader and parser used.

reader

metadata elements

DatabaseReader

  • TableName (only if using the Tables property, omitted if using Query): the fully qualified name of the table to which the record belongs, either in the form <CATALOG>.<SCHEMA>.<TABLE> or <SCHEMA>.<TABLE>, depending on the database

  • OperationName: always SELECT

  • ColumnCount: number of columns in this record

FileReader

  • FileName: fully qualified file name

  • FileOffset: offset in bytes from the beginning of the file to the start of the current event

HDFSReader

  • FileName: Hadoop URL including the fully qualified file name

  • FileOffset: offset in bytes from the beginning of the file to the start of the current event

HTTPReader

  • ClientIPAddress: IP address of the client

  • ClientProtocolVersion: name and version of the protocol the request uses in the form protocol/majorVersion.minorVersion

  • ClientContentLength: length of the request body in bytes, -1 if length is not known.

  • ClientURL: URL the client used to make the request along with the query string

  • Referrer: HTTP header field specified by client that contains the address of the webpage linked to the resource being requested

JMSReader

no metadata returned

KafkaReader

  • TopicName: Kafka topic from which the current event was read

  • PartitionID: Kafka partition from which the current event was read

  • RecordOffset: offset of the current event within the partition

MultiFileReader

  • FileName: fully qualified file name

  • FileOffset: offset in bytes from the beginning of the file to the start of the current event

The following parsers append metadata elements to those of the associated reader:

parser

metadata elements

DSVParser

  • RecordStatus: value is always VALID_RECORD

  • RecordOffset: offset in characters from the beginning of the record to the start of the current event

  • OriginTimeStamp: event origin timestamp

  • RecordEnd: ending character offset of this record in the source

FreeFormTextParser

  • RecordOffset: starting character offset of this record in the source

  • RecordStatus: value is always VALID_RECORD

  • OriginTimeStamp: event origin timestamp

  • RecordEnd: ending character offset of this record in the source

NetflowParser (version 5)

  • version: 5

  • count: number of flows (data records) exported in this packet.

  • sys_uptime: current time in milliseconds since the export device booted

  • unix_secs: current count of seconds since 0000 UTC 1970

  • unix_nsec: residual nanoseconds since 0000 UTC 1970

  • flow_sequence: sequence counter of total flows seen

  • engine_type: type of flow-switching engine

  • engine_id: slot number of the flow-switching engine

  • sampling_interval: first two bits hold the sampling mode, remaining 14 bits hold value of sampling interval

NetflowParser (version 9)

  • Count: total number of records in the Export Packet, which is the sum of Options FlowSet records, Template FlowSet records, and Data FlowSet records

  • ErrorMsg (invalid record only): can be used for debugging

  • package_sequence: incremental sequence counter of all export packets sent from the current observation domain by the exporter

  • Reason (invalid record only): can be used for debugging

  • RecordType: Data (Netflow data record), Template (template details) or Options Template  (information about the Netflow process running in the export device)

  • source_id: 32-bit value that identifies the exporter observation domain

  • SourceIP: source address for packet

  • SourcePort: source port for packet

  • Status: Valid for valid record and template, INVALID for invalid record

  • sys_uptime: time in milliseconds since this export device was first booted

  • Template_ID: ID of the template record (template record is nothing but metdata for a data record) used to decode the data record

  • TemplateStructure (invalid record only): can be used for debugging

  • unix_secs: time in seconds since 0000 UTC 1970, at which the export packet leaves the exporter

  • version: 9

SNMPParser

  • Type: SNMP type

  • Version: SNMP version

  • Community: Community name

  • AgentIp: SNMP agent IP address

  • AgentPort: SNMP agent port no

  • TrapTime: time in milliseconds since the trap was started

  • Enterprise: the management enterprise under whose registration authority the trap was defined

XMLParser

  • RecordOffset: starting character offset of this record in the source

  • RecordStatus: value is always VALID_RECORD

  • OriginTimeStamp: event origin timestamp

  • RecordEnd: ending character offset of this record in the source