Using source and target adapters in applications
An adapter is a process that connects the Striim platform to a specific type of external application or file. Which adapter is selected determines which properties must be specified for a source or target.
For a list of readers, see Readers overview.
For a list of writers, see Writers overview.
Adapter property data types
Adapter properties use the same Supported data types as TQL, plus Encrypted passwords.
Some property data types are enumerated: that is, only documented values are allowed. If setting properties in TQL, be careful not to use other values for these properties.
Connecting with sources and targets over the internet
There are several ways to connect with sources and targets over the internet.
... using cloud provider keys
Some cloud sources and targets, such as Cosmos DB, secure their connections using keys. No additional configuration is required on your part, you simply provide the appropriate key in the source or target properties.
... using an SSH tunnel
See Using an SSH tunnel to connect to a source or target.
... by adding an inbound port rule to your firewall or cloud security group
In the firewall or cloud security group for your source or target, create an inbound port rule for Striim's IP address and the port for your database (typically 3306 for MariaDB or MySQL, 1521 for Oracle, 5432 for PostgreSQL, or 1433 for SQL Server).
To get the IP address for a Striim Cloud service:
In Striim Cloud Console, go to the Services page.
Next to the service, click More and select Security.
Click the Copy IP icon next to the IP address.
... using port forwarding
In your router configuration, create a port forwarding rule for your database's port. If supported by your router, set the source IP to your database's IP address and the target IP to Striim's IP address (which you can get as described above).
Encrypted passwords
Striim encrypts adapter properties of the type com.webaction.security.Password when the adapter is created or altered and decrypts them when providing the values for authentication by a source or target host or service. The cleartext value is not shown in the UI or exported TQL. See also CREATE PROPERTYVARIABLE.
If you are using Oracle JDK 8 or OpenJDK 8 version 1.8.0_161 or later, encryption will be AES-256. With earlier versions, encryption will be AES-128.
To specify a cleartext property in TQL, include Password_encrypted: false
in the adapter properties. This will cause the compiler to encrypt the value when the TQL is loaded.
To encrypt a password for use in TQL, use striim/bin/passwordEncryptor.sh <plaintext password>
. If you are using a Bash or Bourne shell, characters other than letters, numbers, and the following punctuation marks must be escaped: , . _ + : @ % / -
When exporting TQL, you may protect encrypted passwords by specifying a passphrase, which you will need to provide when importing the TQL. This will allow import to a different Striim cluster. Alternatively, you may export without a passphrase, in which case the encrypted passwords in the exported TQL can be decrypted only when imported to the same cluster.
Encrypting other property values
See Using vaults.
Setting rowdelimiter values
Defines the newline string to be used to identify or separate lines when parsing or formatting a file.
\n
(default) is ASCII 010 (line feed / LF), used by UNIX, Linux, and Mac OS X\r
is ASCII 013 (carriage return / CR), used by earlier versions of Mac OS and still used by Excel for Mac when exporting text files\r\n
is CR+LF, used by Windows
Using non-default case and special characters in table identifiers
Striim supports table and column names with non-default case and/or containing special characters in Tables property values for the following databases when read by their own CDC readers, Database Reader, or Incremental Batch Reader and when written to by Database Writer:
MariaDB
MySQL
Oracle Database
Oracle GoldenGate
PostgreSQL
SQL Server
Striim also supports special characters in
Azure Synapse Writer
Snowflake Writer
Supported special characters
character | name | ASCII code | notes |
space | 32 | ||
! | exclamation mark | 33 | |
# | number sign | 35 | not supported in Azure Synapse table names |
$ | dollar | 36 | |
% | percent sign | 37 | not supported in Azure Synapse table names; also see note below |
& | ampersand | 38 | |
( | open parenthesis | 40 | |
) | close parenthesis | 41 | |
+ | plus | 43 | not supported in Azure Synapse table names |
, | comma | 44 | |
- | hyphen | 45 | |
: | colon | 58 | |
; | semicolon | 59 | |
< | less than | 60 | |
= | equals | 61 | |
> | greater than | 62 | |
? | question mark | 63 | not supported in Azure Synapse table names |
@ | at symbol | 64 | |
[ | opening bracket | 91 | not supported for MSJet |
] | closing bracket | 93 | not supported for MSJet |
^ | caret | 94 | |
_ | underscore | 95 | With most sources and targets, this is supported without escaping the name in double quotes. |
{ | opening brace | 123 | |
| | vertical bar | 124 | |
} | closing brace | 125 | |
~ | tilde | 126 |
Notes on using special characters
Identifiers containing special characters must be escaped using double quotes: for example,
MySchema."My@Table"
The following characters must be further escaped as follows:
character
example identifier in database
example escaped in TQL
example escaped in UI
double quote*
ab"c
"ab\\"c"
"ab\"c"
backslash
ab\c
"ab\\\\c"
"ab\\c"
percent*
ab%c
"ab\\%c"
"ab\%c"
*not supported in Azure Synapse table names
In multi-part table names, each identifier containing special characters must be escaped separately: for example,
"My@Schema"."My@Table"
.In three-part table names, special characters are not supported the first part: for example,
MyDB."My@Schema"."My@Table"
.When table names are not escaped, Striim will use the database's default case. For example, If an Oracle table is named MyTable, to read it you must specify the Tables property as
"MyTable"
. If you omit the double quotes, Striim will attempt to read MYTABLE and will fail with an error that the table is not found.Special characters are supported in the ColumnMap function, for example,
ColumnMap("ID"="Pid")
.When replicating data, if the source and target Tables properties use the % wildcard in double quotes (for example,
"My@Schema"."%"
, case and special characters are preserved, provided the WAEvent output of the source is not parsed (see Parsing the data field of WAEvent) before reaching the target. If the output of the source is parsed, special characters will be lost unless stored in a field as strings. For example:CREATE TYPE ParsedDataType( TableName String, ... ); CREATE STREAM OracleTypedStream OF ParsedDataType; CREATE CQ ParseOracleCDCStream INSERT INTO OracleParsedStream SELECT META(x, "TableName").toString(), ... FROM OracleCDCStream x;