ADLS Reader programmer's reference

AccountName

String

The storage account name.

Client ID

String

The Client ID or Application ID of the Microsoft Entra ID application.

Client Secret

Password

The client secret generated for the Microsoft Entra ID application.

Compression Type

String

Set to gzip when the files to be read are in gzip format. Otherwise, leave blank.

Connection Retry Policy

String

retryInterval=30, maxRetries=3

This policy determines how the connection should be retried on a failure. The retryIntervalis the wait time after which the next try to establish connection is attempted.

On exhausting the maxRetries count, the reader will halt the application.

Container

String

The name of the container from which ADLS Reader is expected to read.

Customer Provided Key

Password

The encryption key in a 64-bit String encoding to provide if the Server Side Encryption is based on a Customer Provided Key.

Log Analytics Workspace ID

String

The Log Analytics Workspace ID associated with the Log Analytics workspace that gathers the logs associated with the ADLS Gen2 Storage account.

This property is required only if Object Detection Mode is 'Log Analytics'.

Object Detection Mode

String

ADLS Directory Listing

The mechanism used by ADLS Reader to detect the new objects that are added to the container.

Choose one of these modes:

ADLS Directory Listing: ADLS Reader identifies the changes in each directory by comparing the metadata of the objects in two successive polls.

Log Analytics: ADLS Reader identifies the changes in the entire container based on the log entries over the polling duration using Microsoft Azure Log Analytics. This mode involves comparatively less computational and network overheads. Additional steps have to be performed on the storage account for ADLS Reader to use this object detection mode.

Object Filter

String

*

A wildcard of the object name. Do not modify this property when recovery is enabled for the application.

For example:

*obj.csv

*obj

First-Object*

*

We currently support only the "*" wildcard character.

Polling Interval

Integer

60000 (60 seconds)

The time in milliseconds the adapter will wait between two successive polls to the container to look for object changes. Valid values are between 5000 and 600,000.

Process Sub Folder

Boolean

False

When you enable subfolder processing, the adapter processes objects under the configured path and recursively processes the folders under the configured path.

Start Timestamp

String

Start Timestamp lets you start from a particular point in time. Files modified at or after the configured StartTimeStamp will be processed.

The supported time formats are:

where, YYYY = four-digit year

MM = two-digit month

DD = two-digit day

hh = two digits of hour

mm = two digits of minute

ss = two digits of second

sss = one to three digits representing a decimal fraction of a second

TZD = optional time zone designator (Z or -hh:mm or +hh:mm)

See the Joda time API documentation.

Sub Folder Path

String

The folder path in the container from where the ADLS Reader should exclusively pick the object to be processed. If left empty, the ADLS Reader picks objects from the container's root folder.

Tenant ID

String

The Tenant ID or Directory ID  of the Microsoft Entra ID application.