Skip to main content

Dataverse Reader

Note

This adapter is in preview and is available on Striim Developer only. See Striim Developer for more information.

Microsoft Dataverse lets you securely store and manage data that's used by business applications. The Dataverse Reader is a data integration tool designed to extract and read data from Microsoft Dataverse. It allows users to access a variety of data sources within the Microsoft Dataverse ecosystem.

Feature summary

Feature

Supported?

Notes

Objects

Standard objects

✓

Custom objects

✓

Authentication

Basic authentication

Username and password

OAuth authentication

✓

Manual configuration based

Custom authentication methods

Not all methods may be supported

Operations

Automated mode

✓

Initial load

✓

Pull-based incremental load

✓

Push-based incremental load

Automated pipeline

Governance

Connection profile

Sherlock AI

Sentinel AI

Schema handling

Initial schema creation

✓

Works with supported targets

Schema evolution

Setup

Wizard template

Flow Designer

✓

Striim TQL

✓

Runtime

Resilience/recovery

✓

Parallel execution

Metrics

✓

Standard metrics

Supported authentication

The Dataverse Reader supports connecting to Microsoft Dataverse using OAuth authentication. Creating a connection requires registering the application in Azure AD, setting API permissions for Dataverse, generating a client secret or certificate, obtaining an access token, and obtaining the Dataverse URL.

  1. Register the application in Azure AD.

    1. Create an app in Azure AD to authenticate and interact with resources like Dataverse. See Register an application in Azure AD.

    2. Specify the types of accounts this application should support:

      • For private use applications, select Accounts in this organization directory only.

      • For distributed applications, select one of the multi-tenant options.

      Note

      If you select Accounts in this organizational directory only (default), then you must set the Tenant ID to the ID of the Azure AD Tenant. Otherwise, the authentication attempt fails.

  2. Set the API permissions for Dataverse. Grant the necessary API permissions (for example Dynamics CRM > user_impersonation, Dynamics 365 Business Central > user_impersonation) to interact with Dataverse. See Overview of permissions and consent in the Microsoft identity platform.

  3. Create a client secret under Certificates & secrets for secure authentication.

  4. Obtain the access token needed for the OAuth 2.0 client credentials flow:

    1. Get an access token by sending a POST request to the Azure AD token endpoint using your client ID, client secret, and the Dataverse API scope. See Microsoft identity platform and the OAuth 2.0 client credentials flow.

  5. Obtain the Dataverse URL (organization URL):

    1. To get the Dataverse organization URL, go to the Power Apps portal, select your environment, and find the URL under Settings > Admin Center. Note your Web API endpoint, that is, your serviceRootURL. For example: https://org00000000.crm.dynamics.com/

Supported objects

Dataverse Reader supports reading from all the Tables in the specified Dataverse environment.

Dataverse Reader properties

Property

Type

Default value

Notes

Client ID

String

Client ID of the private app registered in the Active Directory of the Microsoft platform.

Client secret

Password

Client secret of the private app registered in the Active Directory of the Microsoft platform.

Connection pool size

Integer

20

Specifies the maximum number of active connections.

Exclude tables

String

A list of tables excluded from read operations. Typically used to create a list of exceptions when the Tables property includes wildcards. Misconfiguration of the Tables and Exclude Tables properties can cause "Invalid table names" errors.

Incremental load marker

String

The incremental load marker is a unique incremental column in each object used for incremental load. When no marker is specified, tables are resynced at each polling interval.

Specify the name of the column that contains the start position value. This column must meet the following criteria:

  • It should have an integer or timestamp data type (for example, a creation timestamp or an employee ID).

  • The values must be unique and continuously increasing to ensure proper incremental reading.

Migrate schema

Boolean

False

Only available in Initial Load or Automated mode. Set to True to enable initial schema migration, which propagates the object schema from the source to the target.

Mode

Select list:

  • Automated mode

  • Initial load

  • Incremental load

Automated

Automated mode applies incremental updates to objects that support incremental load and performs full resyncs for objects that do not support incremental load.

Organization URL

String

URL endpoint for your Microsoft Dataverse solution.

Polling interval

Integer

5m

Specifies an interval as an integer followed by a unit. Supported units are days (d), hours (h), minutes (m), or seconds (s). The reader polls the source at the specified interval.

Refresh token

Password

An OAuth 2.0 refresh token.Use the value generated while creating the token.

Start Position

String

%=-1

Value of the incremental load marker that defines the initial reading position.

Tables

String

A semicolon-delimited (;) list of objects to read from the source. Supports the % wildcard. Misconfiguration of the Tables and Exclude Tables properties can cause "Invalid table names" errors. Do not modify this property when recovery is enabled for the application.

Tenant ID

String

Tenant Identifier of the directory in which the app is registered.

Thread pool count

Integer

10

The number of parallel running threads. The default value of zero specifies single-threaded operation.

When the value of the thread pool counter is higher than the connection pool size, large data ingestion operations can cause the app to halt. Since best performance is achieved when using one thread for each table being synced, increasing the size of the connection pool to match the number of threads in use is a performance best practice.