Striim 4.0.4 documentation

Creating an app using the Flow Designer

This tutorial uses the sample applications described in Running the CDC demo apps. The Docker PostgreSQL instance must be running. (Kafka is not required.)Running the CDC demo apps

  1. On the Apps page, if ValidatePostgres is not running, deploy and start it.

  2. Select Apps > Create New > Start from scratch, name the app PG2File, select your personal namespace or enter a name for a new namespace, and click Save.

  3. Click the Metadata Browser icon (metadata_browser_icon.png), search for ReadCDCfromPostgresDB, and click Copy to App.

    CreateAppFDMetadataBrowser.png
  4. For Name enter ReadPostgreSQLCDC, for Password enter striim, for New Output enter rawCDCstream, and click Save. Except for the name and output stream, this new source will have the same properties as SamplesDB.ReadCDCfromPostgreDB.

    CreateAppFDNewSource.png
    CreateAppFDNewOutput.png
  5. Click the stream icon, click +, and select Connect next Target component.

    connect_next_component_403.png
  6. Set the properties as follows (leave other properties at their defaults), then click Save.

    • Name: WriteRawData

    • Adapter: FileWriter

    • File Name: RawData

    • Directory: MyDirectory

    • Rollover Policy: leave as is

    • Flush Policy: EventCount 1 (delete Interval)

    • Formatter: JSONFormatter

  7. Deploy and start the application, then return to the View All Apps page and start SamplesDB.Execute250Inserts.

  8. Open Striim\MyDirectory\RawData.00. This contains the raw WAEvent output of PostgreSQLReader formatted as JSON:

     {
      "metadata":{"TableName":"public.customer","TxnID":584,"OperationName":"INSERT" ...
      "data":{
    "C_CUSTKEY":151251,
    "C_NAME":"Customer#151251",
    "C_ADDRESS":"IVhzIApeRb ot,c,E",
    "C_NATIONKEY":15,
    "C_PHONE":"25-989-741-2988",
    "C_ACCTBAL":"711.56",
    "C_MKTSEGMENT":"BUILDING  ",
    "C_COMMENT":"to the even, regular platelets. regular, ironic epitaphs nag e"
    },
      "before":null,
      "userdata":null
     }

    This includes the column names and values for the row inserted into the public.customer table by transaction ID 584.

Modifying an app using the Flow Designer
  1. To parse this raw data, you must write a CQ as described in Parsing the data field of WAEvent. Reopen the PG2File app in Flow Designer, stop and undeploy it, click the stream icon, click +, and select Connect next CQ component.

    ConnectNextComponent2.png
  2. Name the CQ ParseData, name the new output stream ParsedDataStream, and copy and paste the following into the Query field:

    SELECT 
      META(rawCDCstream,"OperationName").toString() AS OpType,
      TO_INT(data[0]) AS CustomerKey,
      TO_STRING(data[1]) AS CustomerName,
      TO_STRING(DATA[2]) AS CustomerAddress,
      TO_INT(DATA[3]) AS NationKey,
      TO_STRING(DATA[4]) AS CustomerPhone,
      TO_DOUBLE(data[5]) AS CustomerAccountBalance,
      TO_STRING(DATA[6]) AS MarketSegment,
      TO_STRING(data[7]) AS CustomerComment
    FROM rawCDCstream;

    Make sure the properties look as shown below, then click Save.

    ParseData.png
  3. Click the lower stream icon, click +, and select Connect next Target component.

    ConnectNextComponent3.png
  4. Set the properties as shown below and click Save.

    • Name: WriteParsedData

    • Adapter: FileWriter

    • File Name: ParsedData

    • Directory: MyDirectory

    • Rollover Policy: leave as is

    • Flush Policy: EventCount 1 (delete Interval)

    • Formatter: JSONFormatter

  5. Deploy and run the application, then return to the Apps page and start SamplesDB.Execute200Inserts.

  6. Open Striim\MyDirectory\ParsedData.00. This contains the parsed data in the custom format defined by the CQ as well as the operation type from the metadata.

    "OpType":"INSERT",
    "CustomerKey":151501,
    "CustomerName":"Customer#151501",
    "CustomerAddress":"IVhzIApeRb ot,c,E",
    "NationKey":15,
    "CustomerPhone":"25-989-741-2988",
    "CustomerAccountBalance":711.56,
    "MarketSegment":"BUILDING  ",
    "CustomerComment":"to the even, regular platelets. regular, ironic epitaphs nag e"