Skip to main content

Introducing automated pipelines

Automated pipeline wizards create two linked applications, one to perform the initial load of existing source data to a target (in most cases using JDBC to read the source) and another to continuously update the target with changes to the source read using CDC. Handover from the first app to the second is handled automatically. Recovery is automatically enabled for both applications, with the exception of initial load for Salesforce sources.

The following is an example of the workflow. This is for creating an on-premise SQL Server to Azure Databricks pipeline, but the steps are similar for all source and target combinations. When you create an automated pipeline, inline help will appear next to the dialogs with detailed information about the settings.

CreateAppMenu.png

Select Apps > Create an App.

CreateAutomatedPipleline01.png

Search for and select your source.

CreateAutomatedPipleline02.png

Click Select Target.

CreateAutomatedPipleline03.png

Search for and select your target.

CreateAutomatedPipleline04.png

Click Get Started. (At this point, you could select initial load only or change data capture only instead of automated pipeline.)

CreateAutomatedPipleline05.png

Name the pipeine and specify the namespace in which to create it, then click Next.

Configure_SQL_Server_Source_with_inline_help.png

Specify the properties for the source as described in the inline help, then click Next..

500_ILCDC_validate_source.png

If validation is successful, the wizard will automatically go on to the next step. If validation fails, click Back and make whatever corrections are required. You must resolve all issues and validate successfully before going forward in the wizard.

500_ILCDC_select_schemas.png

Select the schemas to replicate in the target (do not select system schemas), then click Next.

500_ILCDC_fetch_schemas.png

After Striim has validated and fetched the data, it will automatically go on to the next step.

500_ILCDC_select_tables.png

Select the tables to be replicated on the target (separately for each schema), then click Next.

Configure_Databricks_Target_with_inline_help.png

Specify the properties for the target as described in the inline help, then click Next.

500_ILCDC_validate_target.png

If validation is successful, the wizard will automatically go on to the next step. If validation fails, click Back and make whatever corrections are required. You must resolve all issues and validate successfully before going forward in the wizard.

500_ILCDC_review.png

Review the settings to make sure they are correct, then click Save & Start.

SettingUpYourAutomatedPipeline.png

Striim will create your automated pipeline.

500_ILCDC_IL_progress.png

Once Striim has created the automated pipeline, it will run the initial load app and display its status.

500_ILCDC_handover_complete.png

Once the initial load app has completed, Striim will quiesce it, start the CDC app, and display its status.

500_automated_pipeline_initial_load_error.png

If during either phase the pipeline encounters a problem, it will halt and offer recommendations on what to do to resolve it.

For more information, see Using automated pipeline wizards.