Databricks Writer
Databricks Writer writes to Delta Lake tables in Databricks on AWS or Azure. Delta Lake is an open-source tabular storage framework that includes a transaction log to support features typically associated with relational databases, such as ACID transactions and optimistic concurrency control.
You can use Striim's Databricks Writer to write data from transactional databases such as Oracle and SQL Server, applications such as Salesforce and ServiceNow, NoSQL databases such as Cosmos DB and MongoDB, object stores such as Amazon S3 and Google Cloud Storage, and other supported sources to Delta Lake tables in Databricks on AWS or Azure.
Databricks Writer summary
Supported sources | Databricks Writer can write data from all sources supported by Striim. |
Authentication | Azure Databricks: Databricks Writer authenticates its connection using a personal access token or Microsoft Entra (formerly Azure Active Directory). Databricks on AWS: Databricks Writer authenticates its connection using a personal access token. |
Supported write modes | Databricks Writer supports two write modes:
|
Additional writing features |
|
Supported staging areas | Databricks requires a staging area to temporarily hold new data while it is being written to tables. Databricks Writer supports the following staging areas:
|
Resilience and recovery |
|
Performance | Parallel threads (see Creating multiple writer instances (parallel threads)) can increase throughput to the target in certain situations. |
Programmability |
|
Metrics and auditing | Key metrics are available through Striim's monitoring features (see Monitoring Guide). |
drivers and other third-party libraries | Databricks Writer uses Databricks JDBC driver version 2.6.29. It also uses the following:
|
Key limitations | Data is written in batch mode. Streaming mode is not supported in this release. |
For more information, see:
For Databricks on AWS:
What is Delta Lake?, Databricks on AWS, and Databricks documentation for Amazon Web Services on databricks.com
Databricks on AWS on aws.amazon.com
For Azure Databricks:
Azure Databricks on databricks.com
What is Delta Lake?, Azure Databricks, and Azure Databricks documentation on microsoft.com