Skip to main content

Using the META() function

You can use the META() function to query the metadata map of the WAEvent type, which is used by the output streams of several Striim readers. For example, the following creates a stream containing invalid records:

CREATE STREAM ExceptionStream of ExceptionRecord;
CREATE CQ CQExceptionRecord
INSERT INTO ExceptionStream
SELECT data[0]
FROM CsvStream
WHERE META(CsvStream, ‘RecordStatus’).toString() == ‘INVALID_RECORD’;

The elements of the metadata map vary depending on the reader and parser used.

reader

metadata elements

Database Reader

  • TableName (only if using the Tables property, omitted if using Query): the fully qualified name of the table to which the record belongs, either in the form <CATALOG>.<SCHEMA>.<TABLE> or <SCHEMA>.<TABLE>, depending on the database

  • OperationName: always SELECT

  • ColumnCount: number of columns in this record

File Reader

  • FileName: fully qualified file name

  • FileOffset: offset in bytes from the beginning of the file to the start of the current event

GG Trail Reader

See GG Trail Reader WAEvent fields.

HDFS Reader

  • FileName: Hadoop URL including the fully qualified file name

  • FileOffset: offset in bytes from the beginning of the file to the start of the current event

HP NonStop readers

See HP NonStop reader WAEvent fields.

HTTP Reader

  • ClientIPAddress: IP address of the client

  • ClientProtocolVersion: name and version of the protocol the request uses in the form protocol/majorVersion.minorVersion

  • ClientContentLength: length of the request body in bytes, -1 if length is not known.

  • ClientURL: URL the client used to make the request along with the query string

  • Referrer: HTTP header field specified by client that contains the address of the webpage linked to the resource being requested

JMS Reader

no metadata returned

Kafka Reader

  • TopicName: Kafka topic from which the current event was read

  • PartitionID: Kafka partition from which the current event was read

  • RecordOffset: offset of the current event within the partition

MS SQL Reader / MSJet

See SQL Server readers WAEvent fields.

MariaDB Reader / MySQL Reader

See MySQL Reader WAEvent fields.MySQL Reader WAEvent fields

MultiFile Reader

  • FileName: fully qualified file name

  • FileOffset: offset in bytes from the beginning of the file to the start of the current event

Oracle Reader / OJet

See Oracle Reader and OJet WAEvent fields.

PostgreSQL Reader

See PostgreSQL Reader WAEvent fields.

The following parsers append metadata elements to those of the associated reader:

parser

metadata elements

DSV Parser

  • RecordStatus: value is always VALID_RECORD

  • RecordOffset: offset in characters from the beginning of the record to the start of the current event

  • OriginTimeStamp: event origin timestamp

  • RecordEnd: ending character offset of this record in the source

FreeForm Text Parser

  • RecordOffset: starting character offset of this record in the source

  • RecordStatus: value is always VALID_RECORD

  • OriginTimeStamp: event origin timestamp

  • RecordEnd: ending character offset of this record in the source

Netflow Parser (version 5)

  • version: 5

  • count: number of flows (data records) exported in this packet.

  • sys_uptime: current time in milliseconds since the export device booted

  • unix_secs: current count of seconds since 0000 UTC 1970

  • unix_nsec: residual nanoseconds since 0000 UTC 1970

  • flow_sequence: sequence counter of total flows seen

  • engine_type: type of flow-switching engine

  • engine_id: slot number of the flow-switching engine

  • sampling_interval: first two bits hold the sampling mode, remaining 14 bits hold value of sampling interval

Netflow Parser (version 9)

  • Count: total number of records in the Export Packet, which is the sum of Options FlowSet records, Template FlowSet records, and Data FlowSet records

  • ErrorMsg (invalid record only): can be used for debugging

  • package_sequence: incremental sequence counter of all export packets sent from the current observation domain by the exporter

  • Reason (invalid record only): can be used for debugging

  • RecordType: Data (Netflow data record), Template (template details) or Options Template  (information about the Netflow process running in the export device)

  • source_id: 32-bit value that identifies the exporter observation domain

  • SourceIP: source address for packet

  • SourcePort: source port for packet

  • Status: Valid for valid record and template, INVALID for invalid record

  • sys_uptime: time in milliseconds since this export device was first booted

  • Template_ID: ID of the template record (template record is nothing but metdata for a data record) used to decode the data record

  • TemplateStructure (invalid record only): can be used for debugging

  • unix_secs: time in seconds since 0000 UTC 1970, at which the export packet leaves the exporter

  • version: 9

SNMP Parser

  • Type: SNMP type

  • Version: SNMP version

  • Community: Community name

  • AgentIp: SNMP agent IP address

  • AgentPort: SNMP agent port no

  • TrapTime: time in milliseconds since the trap was started

  • Enterprise: the management enterprise under whose registration authority the trap was defined

XML Parser

  • RecordOffset: starting character offset of this record in the source

  • RecordStatus: value is always VALID_RECORD

  • OriginTimeStamp: event origin timestamp

  • RecordEnd: ending character offset of this record in the source