Skip to main content

Using source and target adapters in applications

Note

Unless otherwise specified, Striim does not support reading from sources and other cloud services or writing to targets and other cloud services whose endpoints are located in AWS GovCloud, Azure Government, or Google Cloud for Government.

An adapter is a process that connects the Striim platform to a specific type of external application or file. Which adapter is selected determines which properties must be specified for a source or target.

For a list of readers, see Readers overview.

For a list of writers, see Writers overview.

Adapter property data types

Adapter properties use the same Supported data types as TQL, plus Encrypted passwords.Encrypted passwords

Some property data types are enumerated: that is, only documented values are allowed. If setting properties in TQL, be careful not to use other values for these properties.

Connecting with sources and targets over the internet

There are several ways to connect with sources and targets over the internet.

... using cloud provider keys

Some cloud sources and targets, such as Cosmos DB, secure their connections using keys. No additional configuration is required on your part, you simply provide the appropriate key in the source or target properties.

... using an SSH tunnel

See Using an SSH tunnel to connect to a source or target.

... by adding an inbound port rule to your firewall or cloud security group

In the firewall or cloud security group for your source or target, create an inbound port rule for Striim's IP address and the port for your database (typically 3306 for MariaDB or MySQL, 1521 for Oracle, 5432 for PostgreSQL, or 1433 for SQL Server).

To get the IP address for a Striim Cloud service:

  1. In Striim Cloud Console, go to the Services page.

  2. Next to the service, click More and select Security.

  3. Click the Copy IP icon next to the IP address.

... using port forwarding

In your router configuration, create a port forwarding rule for your database's port. If supported by your router, set the source IP to your database's IP address and the target IP to Striim's IP address (which you can get as described above).

Encrypted passwords

Striim encrypts adapter properties of the type com.webaction.security.Password using AES-256 when the adapter is created or altered and decrypts them when providing the values for authentication by a source or target host or service. The cleartext value is not shown in the UI or exported TQL. See also CREATE PROPERTYVARIABLE.

To specify a cleartext property in TQL, include Password_encrypted: false in the adapter properties. This will cause the compiler to encrypt the value when the TQL is loaded.

To encrypt a password for use in TQL, use  striim/bin/passwordEncryptor.sh <plaintext password>. If you are using a Bash or Bourne shell, characters other than letters, numbers, and the following punctuation marks must be escaped: , . _ + : @ % / -

When exporting TQL, you may protect encrypted passwords by specifying a passphrase, which you will need to provide when importing the TQL. This will allow import to a different Striim cluster. Alternatively, you may export without a passphrase, in which case the encrypted passwords in the exported TQL can be decrypted only when imported to the same cluster.

Encrypting other property values

See Using vaults.

Setting rowdelimiter values

Defines the newline string to be used to identify or separate lines when parsing or formatting a file.

  • \n (default) is ASCII 010 (line feed / LF), used by UNIX, Linux, and Mac OS X

  • \r is ASCII 013 (carriage return / CR), used by earlier versions of Mac OS and still used by Excel for Mac when exporting text files

  • \r\n is CR+LF, used by Windows

Using non-default case and special characters in table identifiers

Striim supports table and column names with non-default case and/or containing special characters in Tables property values for the following databases when read by their own CDC readers, Database Reader, or Incremental Batch Reader and when written to by Database Writer:

  • MariaDB

  • MySQL

  • Oracle Database

  • Oracle GoldenGate

  • PostgreSQL

  • SQL Server

Striim also supports special characters in

  • Azure Synapse Writer

  • Iceberg Writer (in APPENDONLY mode; in MERGE mode, the only special character supported is _ (underscore))

  • Snowflake Writer

Supported special characters

character

name

ASCII code

notes

space

32

!

exclamation mark

33

#

number sign

35

not supported in Azure Synapse table names

$

dollar

36

%

percent sign

37

not supported in Azure Synapse table names; also see note below

&

ampersand

38

(

open parenthesis

40

)

close parenthesis

41

+

plus

43

not supported in Azure Synapse table names

,

comma

44

-

hyphen

45

:

colon

58

;

semicolon

59

<

less than

60

=

equals

61

>

greater than

62

?

question mark

63

not supported in Azure Synapse table names

@

at symbol

64

[

opening bracket

91

not supported for MSJet

]

closing bracket

93

not supported for MSJet

^

caret

94

_

underscore

95

With most sources and targets, this is supported without escaping the name in double quotes.

{

opening brace

123

|

vertical bar

124

in Iceberg Writer, supported in table names but not column names

}

closing brace

125

~

tilde

126

Notes on using special characters

  • Identifiers containing special characters must be escaped using double quotes: for example, MySchema."My@Table"

  • The following characters must be further escaped as follows:

    character

    example identifier in database

    example escaped in TQL

    example escaped in UI

    double quote*

    ab"c

    "ab\\"c"

    "ab\"c"

    backslash

    ab\c

    "ab\\\\c"

    "ab\\c"

    percent*

    ab%c

    "ab\\%c"

    "ab\%c"

    *not supported in Azure Synapse table names

  • In multi-part table names, each identifier containing special characters must be escaped separately: for example, "My@Schema"."My@Table".

  • In three-part table names, special characters are not supported the first part: for example, MyDB."My@Schema"."My@Table".

  • When table names are not escaped, Striim will use the database's default case. For example, If an Oracle table is named MyTable, to read it you must specify the Tables property as "MyTable". If you omit the double quotes, Striim will attempt to read MYTABLE and will fail with an error that the table is not found.

  • Special characters are supported in the ColumnMap function, for example, ColumnMap("ID"="Pid").

  • When replicating data, if the source and target Tables properties use the % wildcard in double quotes (for example, "My@Schema"."%", case and special characters are preserved, provided the WAEvent output of the source is not parsed (see Parsing the data field of WAEvent) before reaching the target. If the output of the source is parsed, special characters will be lost unless stored in a field as strings. For example:

    CREATE TYPE ParsedDataType(
  TableName String,
  ...
);
CREATE STREAM OracleTypedStream OF ParsedDataType;
CREATE CQ ParseOracleCDCStream
  INSERT INTO OracleParsedStream
  SELECT META(x, "TableName").toString(),
  ...
  FROM OracleCDCStream x;