Salesforce Writer
Writes data to standard or custom Salesforce objects using either single-row or bulk REST APIs. Your Salesforce instance must support access to the API you use.
Limitations:
Changes to a parent object do not affect its child objects.
The maximum integer value supported by Salesforce is 100000000000000000 (100 quadrillion). Any insert of an integer over that value is written as 100000000000000000. Any update of an integer over that value will cause Salesforce Writer to halt with a MALFORMED_QUERY exception.
A Saleforce row error occurs when a batch of data being written succeeds, but Salesforce discards one or more rows. These errors do not generate an exception and the events discarded by Salesforce are not recoverable, though Striim stores them in the application's exception store.
Row errors are typically caused by the following conditions:
invalid data type conversions
duplicate values in unique fields when in APPENDONLY mode
a unique field has a null value
field delimiter characters are in use but unescaped due to Use Quotes being False
Salesforce Writer properties
Property Name | Property Type | Default Value | Description |
---|---|---|---|
API End Point | String | Endpoint of the Salesforce REST API | |
Application Error Count Threshold | Integer | 0 | Application will halt if the number of ignored errors exceeds this number (see Ignorable Error Codes, below). |
Auth Token | encrypted password | If autoAuthTokenRenewal is set to false , specify your Salesforce access token (see Set Up Authorization on developer.salesforce.com: the first section, "Setting Up OAuth 2.0," explains how to create a "connected app"; the second section, "Session ID Authorization," explains how to get the token using If autoAuthTokenRenewal is set to | |
Auto Auth Token Renewal | Boolean | false | With the default value of False, when the specified Auth Token expires the application will halt and you will need to modify it to update the auth token before restarting. This setting is recommended only for development and testing, not in a production environment. Set to True to renew the auth token automatically. In this case, leave Auth Token blank and set the Consumer Key, Consumer Secret, Password, Security Token, and Username properties. |
Batch Policy | String | eventCount:100000, Interval:300 | When Use Bulk API is False, this property is ignored and not shown in the UI. The batch policy includes eventCount and interval (see Setting output names and rollover / upload policiesfor syntax). Events are buffered locally on the Striim server and sent as a batch to the target every time either of the specified values is exceeded. When the app is stopped, any remaining data in the buffer is discarded. To disable batching, set to |
Connection Retry Policy | String | retryInterval = 30 maxRetries = 3 | With the default setting, if a connection attempt is unsuccessful, the adapter will try again in 30 seconds ( |
Consumer Key | String | If autoAuthTokenRenewal is set to true, specify the Consumer Key (see Set Up Authorization on developer.salesforce.com). | |
Consumer Secret | encrypted password | If autoAuthTokenRenewal is set to true, specify the Consumer Secret (see Set Up Authorization on developer.salesforce.com). | |
Field Delimiter | enum | COMMA | When Use Bulk API is False, this property is ignored and not shown in the UI. Supported values: BACKQUOTE, CARET, COMMA, PIPE, SEMICOLON |
Hard Delete | Boolean | False | When Use Bulk API is False, this property is ignored and not shown in the UI. With the default value of False, deleted objects will be moved to Salesforce's recycle bin (see Salesforce Help> Docs > Extend Salesforce with Clicks, Not Code > Manage Deleted Custom Objects). Set to True to bypass the recycle bin and permanently delete the objects. |
Ignorable Error Codes | String | Blank | By default, if Salesforce returns an error, Striim halts the application. Use this property to specify errors (such as INVALID_FIELD) to ignore , separated by commas. For example: IgnorableErrorCode: 'INVALID_FIELD' When you specify an ignorable error, also set the Application Error Count Threshold property to an appropriate number. Ignored exceptions will be written to the application's exception store (see CREATE EXCEPTIONSTORE). |
In Memory | Boolean | True | When Use Bulk API is False, this property is ignored and not shown in the UI. With the default value of True, batches are buffered in memory. Set to False to buffer batches on disk. Use this setting if batches consume too much memory resulting in lower performance or out-of-memory errors. |
JWT Certificate Name | String | When OAuth Authorization Flows is not JWT_BEARER, this property is ignored and does not appear in the UI. | |
JWT Keystore Password | encrypted password | When OAuth Authorization Flows is not JWT_BEARER, this property is ignored and does not appear in the UI. | |
JWT Keystore Path | String | When OAuth Authorization Flows is not JWT_BEARER, this property is ignored and does not appear in the UI. | |
Mode | enum | APPENDONLY | With the default mode of APPENDONLY, update and delete operations in the source are handled as inserts in Salesforce. The input stream's type may be user-defined or WAEvent from a Database Reader, Incremental Batch Reader, or SQL CDC source. Set to MERGE to handle insert and delete operations as inserts and deletes in Salesforce. The input stream's type must be of type WAEvent from a Database Reader, Incremental Batch Reader, or SQL CDC source. The source events must contain at least one field (such as a primary key) that uniquely identifies them, and that field must be mapped to an External ID field in the target object using |
OAuth Authorization Flows | enum | PASSWORD | With the default value of PASSWORD, Salesforce Writer will authorize using OAuth 2.0 username and password (see Salesforce Help> Docs> Identify Your Users and Manage Access > OAuth 2.0 Username-Password Flow for Special Scenarios). In this case, you must specify values for the Consumer Key, Consumer Secret, Password, Security Token, and Username properties. Set to JWT_BEARER to authorize using OAuth 2.0 JWT bearer tokens instead (see Salesforce Help> Docs> Identify Your Users and Manage Access > OAuth 2.0 JWT Bearer Flow for Server-to-Server Integration). In this case, you must specify the Consumer Key, JWT Certificate Name, JWT Keystore Password, JWT Keystore Path, and Username properties. |
Objects | String | See SObjects. | |
Parallel Threads | Integer | When Use Bulk API is False, this property is ignored and not shown in the UI. | |
Password | encrypted password | When autoAuthTokenRenewal is set to true, specify the password for the specified Username (see Encrypted passwords). | |
Security Token | encrypted password | When Auto Auth Token Renewal is set to | |
SObjects | String | In the Flow Designer this property is shown as Objects. The name of an object or objects that Striim will write to. Objects must exist at the time of application start. Multiple objects can be specified when the input stream is replicating from one database to another. Object names can use the wildcard Changes to a parent object do not affect its child objects. If the Salesforce user specified in Username is not an admin, it should have the Read, Create, Edit, Delete, View All, and Modify All permissions. | |
Use Bulk API | Boolean | True | With the default of True, the adapter uses Bulk API 2.0 calls. Incoming records are batched using the specified Batch Policy. We recommend this API for all purposes, particularly:
A single batch may include up to 150MB of data. Each upload uses the following API calls:
You may check the output of the When using the Bulk API, when the Striim application is stopped, any batches already sent to Salesforce will continue processing until their jobs are complete. When set to False, the adapter uses the Force.com REST API and a separate synchronous API call handles each record. We recommend this API only during development, for row-by-row troubleshooting and debugging. |
Use Quotes | Boolean | False | When Use Bulk API is False, this property is ignored and not shown in the UI. With the default value of False, the input must contain no special characers. Set to True to specify that this data contains special characters that must be escaped with double quotes. See Introduction to Bulk API 2.0 and Bulk API / Bulk API 2.0 / Bulk API 2.0 Ingest / Sample CSV Files for details on how the source data must be escaped. |
Username | String | When autoAuthTokenRenewal is set to true, specify an appropriate user name. |
Salesforce Writer sample applications
The following section lists sample TQL code to accomplish several common tasks that involve the Salesforce writer.
Oracle Initial load to Salesforce using auth token only
CREATE SOURCE ORA_SRC1 USING DatabaseReader ( Tables: 'qatest.SRC_TEST', QuiesceOnILCompletion: true, Username: 'qatest', ConnectionURL: 'jdbc:oracle:thin:@localhost:1521:XE', Password: '<password>' ) OUTPUT TO sf1; CREATE TARGET Salesforce_write1 USING SalesforceWriter ( sObjects: 'qatest.SRC_TEST,TGT__c', apiEndPoint: 'https://ap2.salesforce.com', authToken: '<auth_token>' ) INPUT FROM sf1;
Oracle CDC to Salesforce using auth token only
CREATE SOURCE oracle_cdc_src2 USING OracleReader ( ConnectionURL: 'jdbc:oracle:thin:@localhost:1521:XE', Password: '<password>', Username: 'qatest', Tables: 'qatest.CDC1' ) OUTPUT TO sf_cdc2; CREATE TARGET SF_target2 USING SalesforceWriter ( sObjects: 'qatest.CDC1,CDC_TARGET1__c COLUMNMAP(externalID__c=Id)', apiEndPoint: 'https://ap2.salesforce.com', authToken: '<auth_token>', Mode: 'MERGE' ) INPUT FROM sf_cdc2;
Using the Force.com REST API
CREATE TARGET SF_target2 USING SalesforceWriter ( sObjects: 'qatest.CDC1,CDC_TARGET1__c COLUMNMAP(externalID__c=Id)', apiEndPoint: 'https://ap2.salesforce.com', authToken: '<auth_token>', Mode: 'MERGE', useBulkApi: 'false' ) INPUT FROM sf_cdc2;
Using Auth Token renewal
CREATE TARGET SF_target3 USING SalesforceWriter ( autoAuthTokenRenewal: 'true', Username: '<username>', consumerSecret: '<consumer_sercet>', sObjects: 'qatest.CDC1,CDC_TARGET1__c COLUMNMAP(externalID__c=Id)', Password: '<password>', consumerKey: '<consumer_key>', securityToken: '<security_token>', apiEndPoint: 'https://ap2.salesforce.com' ) INPUT FROM sf_cdc3;
Kafka to Salesforce
CREATE STREAM KafkaDSVStream OF WAEvent; CREATE SOURCE KafkaSource USING KafkaReader VERSION '0.11.0' ( brokerAddress:'localhost:9092', Topic:'salesforceTest7', startOffset:0 ) PARSE USING DSVParser () OUTPUT TO KafkaDSVStream; CREATE TYPE AccessLogType ( merchantName String, area string ); CREATE STREAM TypedAccessLogStream OF AccessLogType; CREATE CQ AccesslogCQ INSERT INTO TypedAccessLogStream SELECT data[0], data[1] FROM KafkaDSVStream; CREATE TARGET SF_target2 USING SalesforceWriter ( sObjects: 'kafkatest__c', apiEndPoint: 'https://ap2.salesforce.com', authToken: '<token>', batchpolicy: 'eventcount:4,interval:10s', InMemory: false ) INPUT FROM TypedAccessLogStream;
Salesforce Writer data type support and correspondence
The following apply when the input stream is of a user-defined type.
Striim type | Salesforce data type |
---|---|
java.lang.Object | base64 |
String | boolean |
Byte | byte |
org.joda.time.LocalDate | date |
org.joda.time.DateTime | dateTime |
Double | double |
Long | int |
String | string |
String | time |
The following apply when the input stream is the output of an Oracle Reader source.
Oracle type | Salesforce type |
---|---|
ADT | not supported |
ARRAY | not supported |
BFILE | not supported |
BINARY_DOUBLE | Double |
BINARY_FLOAT | Double |
BFILE | not supported |
BLOB | String |
CHAR | String |
CLOB | String |
DATE | Datetime |
FLOAT | Double |
INTERVALDAYTOSECOND | String |
INTERVALYEARTOMONTH | String |
LONG |
|
LONG RAW | not supported |
NCHAR | String |
NCLOB | String |
NESTED TABLE | not supported |
NUMBER | Double |
NVARCHAR2 | String |
RAW | String |
REF | not supported |
ROWID | String |
TIMESTAMP | Datetime |
TIMESTAMP WITHLOCALTIMEZONE | Datetime |
TIMESTAMP WITHTIMEZONE | Datetime |
UDT | not supported |
UROWID | not supported |
VARCHAR2 | String |
VARRAY | not supported |
XMLTYPE | not supported |