Databricks Writer initial setup

If the bucket does not already exist, create it (see AWS > Documentation > Amazon Simple Storage Service (S3) > User Guide > Create your first S3 bucket).

Select the bucket and click Copy ARN.

Go to the AWS Policy Generator at https://awspolicygen.s3.amazonaws.com/policygen.html

For Select Type of Policy, select IAM Policy.

For AWS Service, select Amazon S3.

Select the individual actions you want to allow or select All Actions.

In the Amazon Resource Name (ARN) field, paste the bucket's ARN that you copied.

Click Add Statement. You should see something similar to this:

Click Generate Policy.

Copy the Policy JSON Document and close the dialog.

Go to the IAM Policies page and click Create policy.

Select the JSON tab, replace the existing text with the policy JSON document you copied, and click Next: Tags > Next: Review.

Enter a descriptive name for the policy (make a note of this as you will need it later), optionally enter a description, and click Create Policy.

Go to the IAM Users page and click Add users.

Enter a name for the IAM user, select Access key, click Next: Permissions.

Select Attach existing policies directly, in Filter policies enter the name of the policy you created, select the policy, and click Next: Tags > Next: Review > Create user.

Click Download .csv. This file contains the access key and secret access key you must specify in Databricks Writer's properties.

Create a general-purpose v2 storage account as described in Learn / Azure / Storage / Blobs / Create a storage account to use with Azure Data Lake Storage Gen2.

Create a service principal and client secret as described in Learn / Tutorial: Connect to Azure Data Lake Storage Gen2

You may manually create the required container, or Databricks Writer can create it automatically, in which case it will be named striim-deltalakewriter-container.

Create a Unity Catalog metastore (see Learn / Azure Databricks documentation / Data Governance / Create a Unity Catalog metastore).

Configure that metastore as described in Learn / Azure Databricks documentation / Data Governance / Create a Unity Catalog metastore / Enable Azure Databricks management for personal staging locations.

Set External Stage Type to PersonalStagingLocation.

Set Personal Staging User Name specify a user name or application ID for a user or service principal with a personal access token (see Learn > Azure Databricks documentation > Administration Guide > Manage personal access tokens):

Log in to the Azure Portal.

Register a new app.

Note the Application ID (referred to as Client ID in this procedure), the OAuth v2 authorization endpoint, and the OAuth v2 token endpoint.

Generate a new Client secret.

Note the Client Secret for future use.

Add the AzureDatabricks API permission.

(When the external stage is ADLS Gen 2) Add the Azure Storage API permission.

Open the following URL in a Web browser.

https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/authorize?
   client_id=<client-id>&
   response_type=code&
   redirect_uri=http%3A%2F%2Flocalhost%3A7734%2Fstriim-callback&
   response_mode=query&
   scope=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d%2F.default%20offline_access

Replace <tenant-id> with with the tenant ID of the registered app. Replace <client-id> with the client ID of the registered app. Provide valid authentication credentials if Azure Portal requests authentication.

The web browser redirects to the specified redirect URI. The authorization code is the part of the URI after the code= string.

Note the authorization code for future use.

Execute the following curl command.

curl -X POST -H 'Content-Type: application/x-www-form-urlencoded' \
   https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token \
   -d 'client_id=<client-id>' \-d 'client_secret=<client_secret>' \
   -d 'scope=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d%2F.default%20offline_access' \
   -d 'code=<authorization_code>' \
   -d 'redirect_uri=http%3A%2F%2Flocalhost%3A7734%2Fstriim-callback' \
   -d 'grant_type=authorization_code'

Replace <tenant-id> with with the tenant ID of the registered app. Replace <client-id> with the client ID of the registered app. Replace <client_secret> with the client secret of the registered app. Replace <authorization_code> with the previously noted authorization code.

The call returns an object that contains an access_token key and a refresh_token key.

Note the value of the refresh_token key.

Open the Postman app.

In the Authorization tab, set the authorization type to OAuth 2.0.

Configure values for the Client ID, Client secret, authorization URL and access token URL.

Set the value of the Scope field to 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default offline_access.

Set the value of the Callback URL field to the redirect URL determined in earlier procedures.

Click Get New Access Token.

Sign into Microsoft Azure and accept the app privilege requests at the consent dialog box.

The browser sends an access token and a refresh token as a response. Note the value of the refresh token.