Parsing the data field of WAEvent
WAEvent is the data type used by the output stream of many readers. Its data
field is an array containing one event's field values. Here is a sample event in WAEvent format from the output stream of CsvDataSource in the PosApp sample application:
WAEvent{ data: ["COMPANY 1159","IQ6wCy3k7PnAiRAN71ROxcNBavvVoUcwp7y","8229344557372754288","1","20130312173212", "0614","USD","329.64","2094770823399082","79769","Odessa"] metadata: {"RecordStatus":"VALID_RECORD","FileName":"posdata.csv","FileOffset":154173} before: null dataPresenceBitMap: "AAA=" beforePresenceBitMap: "AAA=" typeUUID: null };
For information on metadata
, see Using the META() function.
dataPresenceBitMap
, beforePresenceBitMap
, and typeUUID
are reserved and should be ignored.
To parse the data
array, PosApp uses the following TQL:
CREATE CQ CsvToPosData INSERT INTO PosDataStream SELECT TO_STRING(data[1]) AS merchantId, TO_DATEF(data[4],'yyyyMMddHHmmss') AS dateTime, DHOURS(TO_DATEF(data[4],'yyyyMMddHHmmss')) AS hourValue, TO_DOUBLE(data[7]) AS amount, TO_STRING(data[9]) AS zip FROM CsvStream;
PosDataStream
is created automatically using the data types andAS
strings in theSELECT
statement.The order of the
data[#]
functions in theSELECT
clause determines the order of the fields in the output. These may be specified in any order: for example,data[1]
could precededata[0]
.Fields not referenced by the the
SELECT
clause are discarded.The
data[#]
function counts the fields in the array starting from 0, so in this example the first field in the array (COMPANY 1159) is omitted.Non-string values are converted to the types required by the output stream (as defined by the
PosData
type) by theTO_DATEF
,DHOURS
, andTO_DOUBLE
functions (see Functions for more information).
In the PosDataStream output, the parsed version of the sample event shown above is:
merchantId: "IQ6wCy3k7PnAiRAN71ROxcNBavvVoUcwp7y" dateTime: 1363134732000 hourValue: 17 amount: 329.64 zip: "79769"
See PosApp for more information. See also the discussions of ParseAccessLog and ParseLog4J in MultiLogApp for additional examples.
To put the raw, unparsed data
array into a field in a stream, use this syntax:
CREATE CQ CsvRawData INSERT INTO PosRawDataStream SELECT data AS object[] FROM CsvStream;
The field type will be an array of Java objects.