Make the Switch to Efficient, Real-Time ETL
Change Data Capture for Databases
Your business transactions processed in relational databases are critical input to understand the state of your business operations. However, traditionally they are locked in the databases and only accessible through SQL queries. Batch ETL operations using these queries cannot support fast operational decision making or continuous data synchronization needs for hybrid cloud architectures.
Streaming Integration: A Guide to Realizing the Value of Real-Time Data
Change Data Capture with Striim
Collecting this data is only the start, however. To truly work with change data, the whole Striim platform is optimized to work with change, including integrating, processing, delivering, analyzing, and visualizing the changes with Real-Time ETL. Crucially, when applying changes to target databases, or data warehouses, Striim will also respect the ordering and transactionality of those changes.
Striim is the only end-to-end platform to provide ingestion and processing of real-time change data in a single platform, together with analytics, validation, visualization and monitoring.
Moving change data continuously, as new database transactions occur, enables you to analyze up-to-date information and respond to time-sensitive issues immediately. By using Striim’s log-based CDC capabilities you can minimize overhead on the source systems, extending hardware lifetime, and ensure timely data processing without facing batch window limitations.
Moving only the change data, and doing so continuously throughout the day, allows better network bandwidth utilization and drives communication cost savings.
Popular CDC Use Cases
Operational Intelligence
Use real-time change data from transactional systems for operational decision making
Cloud Integration and Migration
Move data between on-premises and cloud systems continuously and without downtime
Replication Monitoring
Monitor any replicated environment to validate database consistency for zero data loss
Striim’s CDC and Real-Time ETL capabilities enable zero data loss for any replicated environment by monitoring the replication process and detecting any latency and consistency issues. The CDC feature, together with Striim’s streaming analytics capabilities, can compare the sources and targets at the transaction level continuously as transactions are replicated.
Striim’s CDC Capabilities
Striim uses log-based CDC to minimize the overhead on the source systems, and reduce the chances of performance degradation. In addition, it is non-intrusive. It does not require changes to the application, such as adding triggers to tables would do. It is a light-weight, but also a highly-performant way to ingest change data. While Striim reads changes from the database logs, these systems continue to run with high-performance for their end users.
Striim’s strengths for real-time CDC are not limited to the ingestion point. The Striim platform includes many additional capabilities to enable robust, end-to-end real-time data integration solutions.
Log-based change data capture
Striim uses log-based change data capture when ingesting from major enterprise databases including Oracle, SQLServer HPE NonStop, MySQL, PostgreSQL, MongoDB, among others. It minimizes CPU overhead on sources, does not require application changes, and substantial management overhead to maintain the solution.
Heterogeneous integration
Striim’s real-time data ingestion is not limited to databases and the CDC method. With Striim you can merge real-time transactional data from OLTP systems with real-time log data, messaging systems, sensor data, NoSQL, and Hadoop data to obtain rich, comprehensive, and reliable information about your business.
Maintains transactional context
When ingesting the change data from database logs, Striim moves committed transactions with the transactional context maintained. Throughout the whole data movement, processing, and delivery steps, this transactional context is preserved so that users can create reliable replica databases.
In-flight transformations
Striim offers out-of-the-box transformers, and in-memory stream processing capabilities to filter, aggregate, mask, transform, and enrich change data while it is in motion. Using SQL-based continuous queries, Striim immediately turns change data into a consumable format for end users, without losing transactional context.
Mission-critical reliability
As the data moves and gets processed through the in-memory components of the Striim platform, every operation is recorded and tracked by the solution. If there is an outage, Striim can replay the transactions from where it was left off — without missing data or having duplicates. Built-in recovery and high-availability using a Striim cluster ensure mission critical reliability.
Built-in data validation
The platform continuously compares the source and target systems, as the data is moving, validating that the databases are consistent and all changed data has been applied to the target. In use cases, where data loss must be avoided, such as migration to a new cloud data store, this feature immensely minimizes migration risks.
Cloud and on-premise change delivery
With the same software, Striim can deliver change data in real time not only to on-premise databases but also to databases running in the cloud, cloud services, messaging systems, files, Hadoop and NoSQL environments. Striim’s integration data flow applications can have multiple targets with concurrent real-time data delivery.
Getting started is easy. Sign up for a free trial or talk to a cloud integration expert.
