Skip to main content

Microsoft 365 Reader

Note

This adapter is in preview and is available on Striim Developer only. See Striim Developer for more information.

The Microsoft 365 Reader is a data integration tool designed to extract and read data from Microsoft Office 365 services. It allows users to access a variety of data sources within the Microsoft 365 ecosystem.

The Microsoft 365 Reader supports incremental loading, meaning it can read only the new or changed data since the last sync, optimizing performance and reducing unnecessary data retrieval. It leverages OAuth authentication for secure access to Microsoft 365 data, ensuring that the integration remains secure and compliant. This adapter is typically used for integrating Microsoft 365 data into business analytics, data lakes, or other enterprise systems, helping organizations gain insights and make data-driven decisions.

Feature summary

Feature

Supported?

Notes

Objects

Standard objects

✓

Custom objects

✓

Authentication

Basic authentication

Username and password

OAuth authentication

✓

Manual configuration based

Custom authentication methods

Not all methods may be supported

Operations

Automated mode

✓

Initial load

✓

Pull-based incremental load

✓

Push-based incremental load

Automated pipeline

Governance

Connection profile

Sherlock AI

Sentinel AI

Schema handling

Initial schema creation

✓

Works with supported targets

Schema evolution

Setup

Wizard template

Flow Designer

✓

Striim TQL

✓

Runtime

Resilience/recovery

✓

Parallel execution

Metrics

✓

Standard metrics

Supported authentication method

Microsoft 365 supports user-based OAuth authentication using Azure Active Directory. Creating a connection for the Microsoft 365 Reader requires you to register a custom application in Azure AD, specify the permissions for that application, and granting administration permissions to operate within an Azure AD tenant.

Authenticating with Azure AD

To create a custom application to authenticate with Azure AD:

  1. Log in to the Microsoft Azure portal.

  2. In the left-hand navigation pane, select Azure Active Directory > App registrations.

  3. Choose New registration.

  4. Enter a name for the application.

  5. Specify the types of accounts this application should support:

    • For private use applications, select Accounts in this organization directory only.

    • For distributed applications, select one of the multi-tenant options.

    Note

    If you select Accounts in this organizational directory only (default), when you establish a connection with the Office 365 Reader you must set AzureTenant to the ID of the Azure AD Tenant. Otherwise, the authentication attempt fails.

  6. Set the redirect URI to http://localhost:33333 (default) or, if you want to specify a different port, specify the desired port and set CallbackURL to the exact reply URL you just defined.

  7. To register the new application, click Register. An application management screen displays. Record these values for later use. (You will use the Application (client) ID value to set the OAuth Client ID parameters, and the Directory (tenant) ID value to set the AzureTenant parameter.)

  8. Navigate to Certificates & Secrets. Select a New Client Secret for this application and specify the desired duration. After the client secret is saved, the Azure App Registration displays the key value. This value is displayed only once, so record it for future use. (You will use it to set the OAuth Client Secret.)

  9. Select the Microsoft Graph API and then select the permissions your app will seek.

  10. If you have specified the use of permissions that require admin consent (such as the Application Permissions), you can grant them from the current tenant on the API Permissions page.

Granting admin consent

Some custom applications require administrative permissions to operate within an Azure Active Directory tenant. Admin consent can be granted when creating a new custom Azure AD application, by adding relevant permissions that are already marked with "Admin Consent Required". Admin consent is also required to use Client Credentials in the authentication flow.

To grant admin consent:

  1. Log in as admin to the Microsoft Azure portal.

  2. Navigate to App Registrations and find the custom Azure AD application you created.

  3. Under API Permissions, choose Grant Consent and follow the wizard.

This gives your application permissions on the tenant under which it was created.

Supported objects

The data in Microsoft 365 has a list of tables in a relational database that can be queried using standard SQL statements. The Microsoft 365 Reader supports reading from the following supported tables:

  • Calendars

  • Contacts

  • Conversations

  • Events

  • Files

  • Groups

  • Messages

  • Tasks

  • Users

Microsoft 365 Reader properties

Property

Type

Default value

Notes

Connection pool size

Integer

20

Specifies the maximum number of active connections.

Exclude tables

String

A list of tables excluded from read operations. Typically used to create a list of exceptions when the Tables property includes wildcards. Misconfiguration of the Tables and Exclude Tables properties can cause "Invalid table names" errors.

Incremental load marker

String

The incremental load marker is a unique incremental column in each object used for incremental load. When no marker is specified, tables are resynced at each polling interval.

Specify the name of the column that contains the start position value. This column must meet the following criteria:

  • It should have an integer or timestamp data type (for example, a creation timestamp or an employee ID).

  • The values must be unique and continuously increasing to ensure proper incremental reading.

Migrate schema

Boolean

False

Only available in Initial Load or Automated mode. Set to True to enable initial schema migration, which propagates the object schema from the source to the target.

Mode

Select list:

  • Automated mode

  • Initial load

  • Incremental load

Automated

Automated mode applies incremental updates to objects that support incremental load and performs full resyncs for objects that do not support incremental load.

OAuth client ID

String

Client ID of the app registered in the Azure Active Directory.

OAuth client secret

Password

Client secret of the app registered in the Azure Active Directory.

OAuth access token

Password

An OAuth 2.0 access token. Use the value generated from the created app.

OAuth refresh token

Password

An OAuth 2.0 refresh token. Use the value generated from the created app.

Polling interval

Integer

5m

Specifies an interval as an integer followed by a unit. Supported units are days (d), hours (h), minutes (m), or seconds (s). The reader polls the source at the specified interval.

Start Position

String

%=-1

Value of the incremental load marker that defines the initial reading position.

Tables

String

A semicolon-delimited (;) list of objects to read from the source. Supports the % wildcard. Misconfiguration of the Tables and Exclude Tables properties can cause "Invalid table names" errors. Do not modify this property when recovery is enabled for the application.

Thread pool count

Integer

10

The number of parallel running threads. The default value of zero specifies single-threaded operation.

When the value of the thread pool counter is higher than the connection pool size, large data ingestion operations can cause the app to halt. Since best performance is achieved when using one thread for each table being synced, increasing the size of the connection pool to match the number of threads in use is a performance best practice.