Resources > eBooks > Streaming Cloud Integration: Key Data Considerations for Hybrid and Multicloud Architectures

Why Striim for Cloud Integration?

Striim offers patented software for streaming data integration across in-house and cloud systems with built-in scalability, security, and reliability. Striim delivers continuous real-time data movement and stream processing across a wide variety of data sources and cloud services on Microsoft Azure, AWS, and Google Cloud platforms. Striim can be deployed in the cloud, on-premises, or both via a hybrid topology.

Striim supports hybrid and multicloud architectures with multiple integration solutions. The key benefits of using Striim for cloud integration include:

  • Moving data in real time via streaming data pipelines to offload operational workloads
  • Ensuring data is delivered in the right format with enriched context to make it immediately actionable
  • Optimizing cloud storage utilization by filtering, aggregating, and transforming data before it reaches cloud targets
  • Reducing risks and disruption of business applications by enabling online or phased data migrations to the cloud
  • Monitoring data streams in real time to prevent data loss
  • Keeping existing in-house systems synchronized by providing real-time data pipelines from cloud environments back to on-prem systems

Core Striim Features for Cloud Integration

The Striim platform offers many unique and robust features to support the real-time data requirements for a wide variety of cloud integration use cases.

Rich Data Sources, Including Enterprise Databases Without Impacting Performance:
Striim ingests real-time data from enterprise databases using low-impact Change Data Capture (CDC). In addition, the Striim platform ingests data from messaging systems (such as Kafka), log files, Hadoop, NoSQL databases, and sensors. With in-memory SQL-based stream processing, Striim enables users to store only the necessary data, in the right format.

Striim enables enterprise databases including Oracle, SQL Server, HPE NonStop, MySQL, PostgreSQL, MongoDB, and Amazon RDS to turn into a streaming data source. Striim further maintains transactional integrity while moving the data in real time, and does not burden the source systems, ensuring high performance for the source production systems. Also, Striim can process the streaming data in-flight using in-memory computing.


Fast Development and Deployment:
Striim’s SQL-based language for filtering, transforming, aggregating, masking, and enriching data empowers a broad set of users to prepare the data for the target system and end users. Striim offers transformer functions via its drag-and-drop UI that enable zero-coding solutions for simple integration use cases. Striim also offers wizards-based PaaS solutions in Azure, Google Cloud, and AWS environments to deliver real-time data pipelines to the cloud in minutes.

Continuous Monitoring and Validation:
With Striim, data pipelines to the cloud can be monitored via live dashboards and a diverse set of metrics. Striim also enables organizations to visualize the streaming data and track business metrics, and allows ad-hoc SQL-based queries on the streaming data. As men- tioned earlier, for database-to-database data movement, Striim offers additional delivery guarantees by tracking the source and target database consistency in real time. Especially in migration use cases, this step helps with moving critical data without data loss, and faster testing of the new environment.

Enterprise-Grade Security, Scalability, and Reliability:
When integrating operational data to the cloud to offload critical high-value workloads, working with an enterprise-grade solution is a must. The bridge that connects the enterprise data center with public clouds must be secure, robust, reliable, scalable, and flexible to support the expansion to public cloud environments. Striim was built from the ground up to handle the demands of high-velocity data in mission-critical environments.

  • Striim is secure with built-in authentication, authorization, protection and encryption. Applications can be secured easily through roles, while specific items can be locked down to the stream level using granular security rules. All data can be encrypted in
  • Striim is high performance and highly scalable with a distributed, modern architecture that combines highly optimized data serialization and windowing with in-memory computing. Striim enables linear scale-out using a low-cost compute infrastructure to support extreme and varied data volumes and
  • Striim is reliable with a fault-tolerant architecture and exactly once processing. Striim comes with a clustered environment that provides built-in high availability and, with advanced checkpointing capabilities, recovery, ensuring no events are missed or processed twice while taking time window contents into account.

Streaming Analytics and Visualization
Striim complements existing operational intelligence solutions with capabilities to analyze and visualize streaming data for time-sensitive insights. Using flexible time series and windowing, Striim performs multi-source, spatial and time-based correlations, complex conditional pattern matching with case statements, predictive analytics, outlier detection, and other statistical operations.

Customers can detect events of interest and gain insight from all their data before it ever lands on disk. Striim stores processed data in-memory for fast access and advanced analytics. Users can run SQL queries on both the streaming data and results data for their ad-hoc and regular reporting needs. Users can also import their own scripts and machine learning models to run on streaming events via the Striim platform.

How Striim Addresses the 5 Use Cases with Hybrid Cloud and Multicloud Architectures

Striim for Cloud Migration

Striim supports overall cloud migration efforts by enabling data and schema migration. With reliable, real-time data movement capabilities, Striim delivers zero database downtime and lowers risks during migration to cloud environments. It supports a wide range of cloud targets on Microsoft Azure, AWS, and Google Cloud, including Azure SQL Database, Azure Cosmos DB, Google Cloud Spanner, Google Cloud SQL, and AWS RDS, to name just a few. Striim combines high-performance initial load with log-based CDC capabilities for all major databases including Oracle, SQL Server, HPE NonStop, MySQL, PostgreSQL, MongoDB, and Amazon RDS.

Striim offers built-in monitoring via live dashboards so that users can see the health of the data pipelines to the cloud. Striim’s CDC offering maintains transactional integrity and comes with end-to-end reliability without data loss or duplicates (exactly once processing). Also, Striim’s CDC capabilities enable data delivery validation that supports zero data loss during the migration process to minimize risks. Striim enables parallelization when delivering to databases and Kafka solutions to effectively handle high data volumes.

By keeping the old and new systems in sync indefinitely, Striim enables extensive testing of the new environment – using production data and without time limitation – and further minimizes cloud migration risks. Once the cutover has been accomplished, users can apply the transactions that took place in the new database back to the old system. As a result, users can cutover back to the original system if needed.

In addition to delivering enterprise-grade and easy-to-design real-time data pipelines, Striim can be integrated with existing monitoring solutions to automate end-to-end migration. This capability allows performing migrations at scale according to established migration schedules.

Striim combines bulk initial load with low-impact CDC and real-time data delivery validation to minimize downtime and data loss risks, and provide exactly once processing.

Striim for Building New Business Systems in the Cloud

With real-time data ingestion capabilities from enterprise databases, messaging systems (including Kafka), sensors, and log files, Striim eases hybrid cloud architectures for operational systems. Striim can continuously feed real-time transactions and events from in-house and cloud-based data stores to Azure Event Hubs, Google Pub/Sub, Amazon Kinesis, as well as Azure SQL, Google Cloud SQL, Amazon RDS and Aurora databases in real-time.

For enterprise database sources, Striim uses log-based CDC to avoid performance degradation for source systems. Striim’s CDC does not require changes to the source system, either.

While moving data continuously in its in-memory platform, Striim maintains transaction integrity and can perform encryption and masking on streaming data to help comply with privacy regulations.

With built-in high-availability and checkpointing capabilities, Striim provides out-of-the-box exactly once processing, meaning no data duplicates and no data loss, even in the case of outages. Designed for mission-critical environments, Striim customers heavily rely on its secure, reliable, and fault-tolerant distributed architecture. Striim’s built-in scale-out architecture makes it easy for customers to handle extremely large data volumes by adding new resources as required.

Striim enables real-time data distribution to cloud-based messaging systems, databases and storage solutions.

Striim for Cloud-Based Analytics

Striim’s real-time data ingestion capability goes beyond enterprise databases and includes messaging systems (such as Kafka), log files, sensors, Hadoop, and NoSQL systems, so users can use any data they need for advanced analytics in the cloud. Striim is also architected for maintaining sub-second latency even in high-data-volume environments. With a scale-out architecture, users can keep up with growing data volumes using inexpensive hardware.

Striim processes in-flight data in-memory using SQL-based stream processing, including filtering, aggregation, masking, transformations, denormalization, and enrichment, with both dynamically changing and static data. In addition to preparing the data for analytics, Striim can perform correlations, pattern matching, and anomaly detection on data-in-motion to provide time-sensitive insights before the data is delivered to the cloud.

Interactive, real-time dashboards enable users to not only monitor the health of the integration processes, but also the content of the data. Striim users are able to build their dashboards in minutes to track key business metrics on live, streaming business data, brought together from a wide range of sources.

Striim continuously loads real-time data to cloud-based reporting and analytics solutions in a consumable format.

Striim for Multicloud Integration

Striim supports multiple cloud vendors and can simultaneously stream data from on-premises systems to a wide range of cloud targets on Google Cloud, Microsoft Azure, and AWS. Striim can deliver to multiple targets in the same integration application, reducing solution complexity. In addition, it can ingest multiple data streams in a single integration application, and merge them as needed before delivering to a single or multiple cloud targets.

Striim supports databases, data warehouses, messaging, storage and big data solutions on AWS, Azure, and GCP.

Striim can run as a PaaS solution with subscription pricing on the Azure, Google Cloud, and AWS marketplaces, with the ability to deliver data to all the cloud services shown in the diagram above.

Striim also provides comprehensive data pipeline monitoring across cloud and on-premises environments with real-time data shown in interactive dashboards.

Striim for Inter-Cloud Integration

In addition to delivering real-time data to multiple cloud vendors with pay-as-you-go PaaS solutions (see the previous section), Striim can ingest real-time data from databases, messaging systems, Hadoop and NoSQL solutions running on the AWS, Azure and GCP infrastructures; PaaS offerings such as AWS RDS, AWS S3, Azure HDInsight; and SaaS applications such as Salesforce.

With its ability to deliver pre-processed data to databases, data warehouses, files, messaging solutions, Hadoop, and NoSQL environments, Striim can distribute real-time cloud data to on-premises private cloud environments as well.

How Striim’s Customers Gain Greater Value from Cloud

Striim has helped several industry leaders in their cloud journey by providing streaming data pipelines across existing and new cloud solutions. Here are a couple of examples that illustrate the value of streaming data integration for cloud adoption.

North American Financial Services Company

With the decision to adopt a cloud-first strategy, this leading North American bank needed to build real-time data pipelines from its existing on-premises and cloud systems to its new, preferred cloud environment. As part of its commitment to offloading workloads to the cloud, the bank decided to move their financial reporting system from an on-premises enterprise database to Microsoft Azure.

Striim was initially used to load the existing financial transactions, and thereafter continuously captures real-time change data from the financial systems on HPE NonStop and the opera- tional reporting database on Oracle Database – non-intrusively. It delivers the streaming data to an Azure Event Hubs environment in the cloud that supports the reporting application.

Using Striim as a PaaS solution, the bank can rapidly deploy real-time data pipelines to Azure to support its cloud-first strategy, and provide bank employees with 24/7 access to current financial information. In addition, Striim directly streams data from its transactional systems into the cloud without impacting source systems, and enables the bank to avoid maintaining a separate on-premises database for reporting.

By using a streaming data integration solution for its cloud environment, the bank can make critical risk management and other operational decisions based on real-time data, as compared to their old method of using day-old data.

Leading International Airport

This leading international airport uses Striim to become a world-class hub on the West coast by streamlining its operations and enhancing the passenger experience. By combining log-based change data capture with bulk loading from enterprise databases, Striim keeps the airport’s cloud-based analytical solutions up-to-date with passenger and flight information.

Striim delivers real-time data to both Azure SQL Database and to Snowflake data warehouse to support daily operational decision making such as ticketing, flight schedules, luggage handling, and wheelchair operations. Striim also provides real-time analytics and dashboards to help the airport with staff optimization and unique services for passengers.

Striim analyzes streaming passenger and flight data to help the airport identify passengers at risk of missing connections. Also, Striim issues alerts on real-time flight schedule changes and higher-than-expected passenger loads. With this information, the airport can optimize its staff and reduce the lines for customs and immigration.

Conclusion

The agility and modernization benefits of digital transformation depend on running business-critical applications and systems in the cloud. Real-time data integration is crucial to cloud adoption, which requires a solid bridge that enables secure, reliable, and scalable data connectivity. Continuous and timely data flow from on-premises to cloud, or from incumbent cloud solutions to a new cloud environment, is a key requirement for businesses to offload their high-value, critical workloads to the cloud.

Streaming integration enables businesses to gain maximum operational value while easing the transition to the cloud by minimizing business interruptions and risks. It lowers risks of data migration to the cloud and between clouds, enables real-time data distribution to new business systems in the cloud, provides a smart data architecture for optimized cloud storage, and delivers timely, pre-processed data for operational intelligence from cloud analytics.

Striim is uniquely qualified to help make the journey to cloud fast and smooth. Striim simplifies moving real-time data across a diverse set of data sources and cloud targets with high-performance, security, and reliability. With a patented design to support business- critical systems with high data volumes, Striim delivers reliable, continuous, and secure access to real-time data in the cloud.