Striim 3.9.8 documentation

Creating an app using the Flow Designer

This tutorial uses the sample applications described in Running the CDC sample apps. The Docker PostgreSQL instance must be running. (Kafka is not required.)

  1. On the Apps page, if ValidatePostgres is not running, deploy and start it.

  2. Select Add App > Start from Scratch, name the app PG2File, select your personal namespace or enter a name for a new namespace, and click Save.

  3. Open the Metadata Browser, open the Source folder, hover the mouse over ReadCDCfromPostgreDB, and click Copy to App.

    MetadataBrowser.png
  4. Name the source ReadPostgreSQLCDC, name the new output stream rawCDCstream, and click Save. Except for the name and output stream, this new source will have the same properties as SamplesDB.ReadCDCfromPostgreDB.

    ReadPostgreSQLCDC.png
  5. Hover the mouse over the stream icon, click +, and select Connect next Target component.

    ConnectNextComponent1.png
  6. Set the properties as shown below and click Save.

    WriteRawData2.png
  7. Deploy and run the application, then return to the Apps page and start SamplesDB.Execute200Inserts.

  8. Open Striim\MyDirectory\RawData.00. This contains the raw WAEvent output of PostgreSQLReader formatted as JSON:

     {
      "metadata":{"TableName":"public.customer","TxnID":584,"OperationName":"INSERT" ...
      "data":{
    "C_CUSTKEY":151251,
    "C_NAME":"Customer#151251",
    "C_ADDRESS":"IVhzIApeRb ot,c,E",
    "C_NATIONKEY":15,
    "C_PHONE":"25-989-741-2988",
    "C_ACCTBAL":"711.56",
    "C_MKTSEGMENT":"BUILDING  ",
    "C_COMMENT":"to the even, regular platelets. regular, ironic epitaphs nag e"
    },
      "before":null,
      "userdata":null
     }

    This includes the column names and values for the row inserted into the public.customer table by transaction ID 584.

  9. To parse this raw data, you must write a CQ as described in Parsing the data field of WAEvent. Reopen the PG2File app in Flow Designer, hover the mouse over the stream icon, click +, and select Connect next CQ component.

    ConnectNextComponent2.png
  10. Name the CQ ParseData, name the new output stream ParsedDataStream, and copy and paste the following into the Query field:

    SELECT 
      META(rawCDCstream,"OperationName").toString() AS OpType,
      TO_INT(data[0]) AS CustomerKey,
      TO_STRING(data[1]) AS CustomerName,
      TO_STRING(DATA[2]) AS CustomerAddress,
      TO_INT(DATA[3]) AS NationKey,
      TO_STRING(DATA[4]) AS CustomerPhone,
      TO_DOUBLE(data[5]) AS CustomerAccountBalance,
      TO_STRING(DATA[6]) AS MarketSegment,
      TO_STRING(data[7]) AS CustomerComment
    FROM rawCDCstream;

    Make sure the properties look as shown below, then click Save.

    ParseDataCQ2.png
  11. Hover the mouse over the lower stream icon, click +, and select Connect next Target component.

    ConnectNextComponent3.png
  12. Set the properties as shown below and click Save.

    WriteParsedData2.png
  13. Deploy and run the application, then return to the Apps page and start SamplesDB.Execute200Inserts.

  14. Open Striim\MyDirectory\ParsedData.00. This contains the parsed data in the custom format defined by the CQ as well as the operation type from the metadata.

    "OpType":"INSERT",
    "CustomerKey":151501,
    CustomerName":"Customer#151501",
    "CustomerAddress":"IVhzIApeRb ot,c,E",
    "NationKey":15,
    "CustomerPhone":"25-989-741-2988",
    "CustomerAccountBalance":711.56,
    "MarketSegment":"BUILDING  ",
    "CustomerComment":"to the even, regular platelets. regular, ironic epitaphs nag e"