Skip to main content

Salesforce Writer

Writes data to standard or custom Salesforce objects using either single-row or bulk REST APIs. Your Salesforce instance must support access to the API you use.

Limitations:

  • Changes to a parent object do not affect its child objects.

  • The maximum integer value supported by Salesforce is 100000000000000000 (100 quadrillion). Any insert of an integer over that value is written as 100000000000000000. Any update of an integer over that value will cause Salesforce Writer to halt with a MALFORMED_QUERY exception.

  • A Saleforce row error occurs when a batch of data being written succeeds, but Salesforce discards one or more rows. These errors do not generate an exception and the events discarded by Salesforce are not recoverable, though Striim stores them in the application's exception store.

    Row errors are typically caused by the following conditions:

    • invalid data type conversions

    • duplicate values in unique fields when in APPENDONLY mode

    • a unique field has a null value

    • field delimiter characters are in use but unescaped due to Use Quotes being False

Salesforce Writer properties

Property Name

Property Type

Default Value

Description

API End Point

String

Endpoint of the Salesforce REST API

Application Error Count Threshold

Integer

0

Application will halt if the number of ignored errors exceeds this number (see Ignorable Error Codes, below).

Auth Token

encrypted password

If autoAuthTokenRenewal is set to false , specify your Salesforce access token (see Set Up Authorization on developer.salesforce.com: the first section, "Setting Up OAuth 2.0," explains how to create a "connected app"; the second section, "Session ID Authorization," explains how to get the token using curl).

If autoAuthTokenRenewal is set to true, leave blank.

Auto Auth Token Renewal

Boolean

false

With the default value of False, when the specified Auth Token expires the application will halt and you will need to modify it to update the auth token before restarting. This setting is recommended only for development and testing, not in a production environment. When this property is False, you must specify Auth Token, Password, and Username.

Set to True to renew the auth token automatically. In this case, leave Auth Token blank and set the Consumer Key, Consumer Secret, Password, Security Token, and Username properties.

Batch Policy

String

eventCount:100000, Interval:300

When Use Bulk API is False, this property is ignored and not shown in the UI.

The batch policy includes eventCount and interval (see Setting output names and rollover / upload policies for syntax). Events are buffered locally on the Striim server and sent as a batch to the target every time either of the specified values is exceeded. When the app is stopped, any remaining data in the buffer is discarded. To disable batching, set to EventCount:1,Interval:0.

Connection Retry Policy

String

retryInterval = 30

maxRetries = 3

With the default setting, if a connection attempt is unsuccessful, the adapter will try again in 30 seconds (retryInterval. If the second attempt is unsuccessful, in 30 seconds it will try a third time (maxRetries). If that is unsuccessful, the adapter will fail and log an exception. Negative values are not supported.

Consumer Key

String

If Auto Auth Token Renewal is set to true, specify the Consumer Key (see Set Up Authorization on developer.salesforce.com).

Consumer Secret

encrypted password

If Auto Auth Token Renewal is set to true, specify the Consumer Key (see Set Up Authorization on developer.salesforce.com).

Field Delimiter

enum

COMMA

When Use Bulk API is False, this property is ignored and not shown in the UI.

Supported values: BACKQUOTE, CARET, COMMA, PIPE, SEMICOLON

Hard Delete

Boolean

False

When Use Bulk API is False, this property is ignored and not shown in the UI.

With the default value of False, deleted objects will be moved to Salesforce's recycle bin (see Salesforce Help> Docs > Extend Salesforce with Clicks, Not Code > Manage Deleted Custom Objects).

Set to True to bypass the recycle bin and permanently delete the objects.

Ignorable Error Codes

String

Blank

By default, if Salesforce returns an error, Striim halts the application. Use this property to specify errors (such as INVALID_FIELD) to ignore , separated by commas. For example:

IgnorableErrorCode: 'INVALID_FIELD'

When you specify an ignorable error, also set the Application Error Count Threshold property to an appropriate number.

Ignored exceptions will be written to the application's exception store (see CREATE EXCEPTIONSTORE).

In Memory

Boolean

True

When Use Bulk API is False, this property is ignored and not shown in the UI.

With the default value of True, batches are buffered in memory.

Set to False to buffer batches on disk. Use this setting if batches consume too much memory resulting in lower performance or out-of-memory errors.

JWT Certificate Name

String

When OAuth Authorization Flows is not JWT_BEARER, this property is ignored and does not appear in the UI.

See Salesforce Help> Docs> Identify Your Users and Manage Access > OAuth 2.0 JWT Bearer Flow for Server-to-Server Integration.

JWT Keystore Password

encrypted password

When OAuth Authorization Flows is not JWT_BEARER, this property is ignored and does not appear in the UI.

See Salesforce Help> Docs> Identify Your Users and Manage Access > OAuth 2.0 JWT Bearer Flow for Server-to-Server Integration.

JWT Keystore Path

String

When OAuth Authorization Flows is not JWT_BEARER, this property is ignored and does not appear in the UI.

See Salesforce Help> Docs> Identify Your Users and Manage Access > OAuth 2.0 JWT Bearer Flow for Server-to-Server Integration.

Mode

enum

APPENDONLY

With the default mode of APPENDONLY, update and delete operations in the source are handled as inserts in Salesforce. The input stream's type may be user-defined or WAEvent from a Database Reader, Incremental Batch Reader, or SQL CDC source.

Set to MERGE to handle insert and delete operations as inserts and deletes in Salesforce. The input stream's type must be of type WAEvent from a Database Reader, Incremental Batch Reader, or SQL CDC source. The source events must contain at least one field (such as a primary key) that uniquely identifies them, and that field must be mapped to an External ID field in the target object using ColumnMap in the Tables property. If an External ID matches an object in the target, it will be updated. If an External ID field is not present in the Salesforce object, Salesforce Writer will halt.

OAuth Authorization Flows

enum

PASSWORD

With the default value of PASSWORD, Salesforce Writer will authorize using OAuth 2.0 username and password (see Salesforce Help> Docs> Identify Your Users and Manage Access > OAuth 2.0 Username-Password Flow for Special Scenarios). In this case, you must specify values for the Consumer Key, Consumer Secret, Password, Security Token, and Username properties.

Set to JWT_BEARER to authorize using OAuth 2.0 JWT bearer tokens instead (see Salesforce Help> Docs> Identify Your Users and Manage Access > OAuth 2.0 JWT Bearer Flow for Server-to-Server Integration). In this case, you must specify the Consumer Key, JWT Certificate Name, JWT Keystore Password, JWT Keystore Path, and Username properties.

Objects

String

See SObjects.

Parallel Threads

Integer

When Use Bulk API is False, this property is ignored and not shown in the UI.

See Creating multiple writer instances.

Password

encrypted password

When Auto Auth Token Renewal is set to true, specify the password for the specified Username (see Encrypted passwords).

Security Token

encrypted password

When Auto Auth Token Renewal is set to true, specify the security token for the specified username (see Reset Your Security Token on help.salesforce.com).

SObjects

String

In the Flow Designer this property is shown as Objects.

The name of an object or objects that Striim will write to. Objects must exist at the time of application start. Multiple objects can be specified when the input stream is replicating from one database to another. Object names can use the wildcard % and are case insensitive.

Changes to a parent object do not affect its child objects.

If the Salesforce user specified in Username is not an admin, it should have the Read, Create, Edit, Delete, View All, and Modify All permissions.

Use Bulk API

Boolean

True

With the default of True, the adapter uses Bulk API 2.0 calls. Incoming records are batched using the specified Batch Policy. We recommend this API for all purposes, particularly:

  • when you have more than 1000 events per second

  • when batching is desirable

  • when you want Hard Delete or Parallel Threads

A single batch may include up to 150MB of data. Each upload uses the following API calls:

  1. /services/data/v<XX.X>/jobs/ingest creates a job.

  2. /services/data/v<XX.X>/jobs/ingest/<jobID>/batches uploads job data.

  3. /services/data/v<XX.X>/jobs/ingest/<jobID>/ closes a job and submits the data for processing.

You may check the output of the /services/data/v<XX.X>/jobs/ingest/<jobID>/successfulResults/ and /services/data/v<XX.X>/jobs/ingest/<jobID>/failedResults/ API calls to determine the status of a given job.

When using the Bulk API, when the Striim application is stopped, any batches already sent to Salesforce will continue processing until their jobs are complete.

When set to False, the adapter uses the Force.com REST API and a separate synchronous API call handles each record. We recommend this API only during development, for row-by-row troubleshooting and debugging.

Use Quotes

Boolean

False

When Use Bulk API is False, this property is ignored and not shown in the UI.

With the default value of False, the input must contain no special characers.

Set to True to specify that this data contains special characters that must be escaped with double quotes. See Introduction to Bulk API 2.0 and Bulk API / Bulk API 2.0 / Bulk API 2.0 Ingest / Sample CSV Files for details on how the source data must be escaped.

Username

String

When autoAuthTokenRenewal is set to true, specify an appropriate user name.

Salesforce Writer sample applications

The following section lists sample TQL code to accomplish several common tasks that involve the Salesforce writer.

Oracle Initial load to Salesforce using auth token only

CREATE SOURCE ORA_SRC1 USING DatabaseReader (
  Tables: 'qatest.SRC_TEST',
  QuiesceOnILCompletion: true,
  Username: 'qatest',
  ConnectionURL: 'jdbc:oracle:thin:@localhost:1521:XE',
  Password: '<password>'
  )
OUTPUT TO sf1;

CREATE TARGET Salesforce_write1 USING SalesforceWriter
(  
  sObjects: 'qatest.SRC_TEST,TGT__c',
  apiEndPoint: 'https://ap2.salesforce.com',
  authToken: '<auth_token>'
  )
INPUT FROM sf1;

Oracle CDC to Salesforce using auth token only

CREATE SOURCE oracle_cdc_src2 USING OracleReader (
  ConnectionURL: 'jdbc:oracle:thin:@localhost:1521:XE',
  Password: '<password>',
  Username: 'qatest',
  Tables: 'qatest.CDC1'
)
OUTPUT TO sf_cdc2;

CREATE TARGET SF_target2 USING SalesforceWriter (
  sObjects: 'qatest.CDC1,CDC_TARGET1__c COLUMNMAP(externalID__c=Id)',
  apiEndPoint: 'https://ap2.salesforce.com',
  authToken: '<auth_token>',
  Mode: 'MERGE'
)
INPUT FROM sf_cdc2;

Using the Force.com REST API

CREATE TARGET SF_target2 USING SalesforceWriter (
  sObjects: 'qatest.CDC1,CDC_TARGET1__c COLUMNMAP(externalID__c=Id)',
  apiEndPoint: 'https://ap2.salesforce.com',
  authToken: '<auth_token>',
  Mode: 'MERGE',
  useBulkApi: 'false'
)
INPUT FROM sf_cdc2;

Using Auth Token renewal

CREATE TARGET SF_target3 USING SalesforceWriter (
  autoAuthTokenRenewal: 'true',  
  Username: '<username>',
  consumerSecret: '<consumer_sercet>',
  sObjects: 'qatest.CDC1,CDC_TARGET1__c COLUMNMAP(externalID__c=Id)',
  Password: '<password>',
  consumerKey: '<consumer_key>',
  securityToken: '<security_token>',
  apiEndPoint: 'https://ap2.salesforce.com'
)
INPUT FROM sf_cdc3;

Kafka to Salesforce

CREATE STREAM KafkaDSVStream OF WAEvent;
CREATE SOURCE KafkaSource USING KafkaReader VERSION '0.11.0' (
  brokerAddress:'localhost:9092',
  Topic:'salesforceTest7',
  startOffset:0
)
PARSE USING DSVParser ()
OUTPUT TO KafkaDSVStream;

CREATE TYPE AccessLogType (
  merchantName String,
  area string
);
CREATE STREAM TypedAccessLogStream OF AccessLogType;

CREATE CQ AccesslogCQ
INSERT INTO TypedAccessLogStream
SELECT data[0],
data[1]
FROM KafkaDSVStream;

CREATE TARGET SF_target2 USING SalesforceWriter (
  sObjects: 'kafkatest__c',
  apiEndPoint: 'https://ap2.salesforce.com',
  authToken: '<token>',
  batchpolicy: 'eventcount:4,interval:10s',
  InMemory: false
)
INPUT FROM TypedAccessLogStream;

Salesforce Writer data type support and correspondence

The following apply when the input stream is of a user-defined type.

Striim type

Salesforce data type

java.lang.Object

base64

String

boolean

Byte

byte

org.joda.time.LocalDate

date

org.joda.time.DateTime

dateTime

Double

double

Long

int

String

string

String

time

The following apply when the input stream is the output of an Oracle Reader source.

Oracle type

Salesforce type

ADT

not supported

ARRAY

not supported

BFILE

not supported

BINARY_DOUBLE

Double

BINARY_FLOAT

Double

BFILE

not supported

BLOB

String

CHAR

String

CLOB

String

DATE

Datetime

FLOAT

Double

INTERVALDAYTOSECOND

String

INTERVALYEARTOMONTH

String

LONG

  • for Oracle Reader, results may be inconsistent, Oracle recommends using CLOB instead

  • for OJet, String

LONG RAW

not supported

NCHAR

String

NCLOB

String

NESTED TABLE

not supported

NUMBER

Double

NVARCHAR2

String

RAW

String

REF

not supported

ROWID

String

TIMESTAMP

Datetime

TIMESTAMP WITHLOCALTIMEZONE

Datetime

TIMESTAMP WITHTIMEZONE

Datetime

UDT

not supported

UROWID

not supported

VARCHAR2

String

VARRAY

not supported

XMLTYPE

not supported