Implementing Gartner’s Cloud Smart FEVER selection process using Striim

In their recent research note, “Move From Cloud First to Cloud Smart to Improve Cloud Journey Success” (February 2020), Gartner introduced the concept of using the FEVER selection process to prioritize workloads to move to cloud.

According to the research note, to ensure rapid results by building on the knowledge of earlier experiences with cloud, IT leaders “should prioritize the workloads to move to cloud by using a ‘full circle’ continuous loop selection process: faster, easier, valuable, efficient and repeat (FEVER; see Figure 2). This allows them to deliver results in waves of migrations according to the organization’s delivery capacities.”

While thinking about this concept I realized that following this approach is one of the reasons that Striim’s customers are so successful with their cloud migration and integration initiatives.  They are utilizing a cloud smart approach for real-world use-cases, including online database migrations enabled by change data capture, offloading reporting to cloud environments, and continuous data delivery for cloud analytics.

Faster

The speed of solutions is critical to many of our customers that have strict SLAs, and limited timeframes in which they want to complete their projects. Striim allows customers to build and test data flows supporting cloud adoption very quickly, while Striim’s optimized architecture enables rapid transfer of data from data sources to cloud for both initial load, and on-going real-time data delivery.

Easier

Customers don’t want to spend days or weeks learning a new solution. In order to implement quickly, the solution must be easy to learn and work with. Striim’s wizard-based approach and intuitive UI enables our customers to rapidly build out their data pipelines, and transfer knowledge for on-going operations.

Valuable

Many of our customers are already ‘Cloud Smart’ and approach cloud initiatives in a pragmatic way. They often start with highly critical, but simple migrations, that gives them the highest value in the shortest time. Once all the “lowest-hanging fruits” are picked and successfully implemented, they move onto more complex scenarios, or integrate additional sources.

Efficient

Cost-efficiency for our customers is more than just the on-going cost reductions inherent in moving to a cloud solution. It also includes the time taken by their valuable employees to build and maintain the solution, and the data ingress costs inherent in moving their data to the cloud. By utilizing Striim, they can reduce the amount of time spent to achieve success and reduce their data movement costs by utilizing one-time loads, with on-going change delivery.

Repeat

It is seldom that our customers have a single migration, or cloud adoption to perform. Repeatability, and reusability of the cloud migration or integration is essential to their long-term plans. Not only do they want to be able to repeat similar migrations, but they also want to be able to use the same platform for all of their cloud adoption initiatives. By standardizing on Striim, our customers can take advantage of the large numbers of sources and cloud targets we support and focus on the business imperatives without having to worry whether it’s possible.

 

If you would like to learn more about becoming cloud smart, you can access the full report “Move From Cloud First to Cloud Smart to Improve Cloud Journey Success” (February 2020), for a limited time using this link.

 

Move From Cloud First to Cloud Smart to Improve Cloud Journey Success, Henrique Cecci, 25 February 2020

GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally, and is used herein with permission. All rights reserved.

This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available upon request from Striim.

Cloud Adoption: How Streaming Integration Minimizes Risks

 

 

Last week, we hosted a live webinar, Cloud Adoption: How Streaming Integration Minimizes Risks. In just 35 minutes, we discussed how to eliminate database downtime and minimize other risks of cloud migration and ongoing integration for hybrid cloud architecture, including a live demo of Striim’s solution.

Our first speaker, Steve Wilkes, started the presentation discussing the importance of cloud adoption for today’s pandemic-impacted, fragile business environment. He continued with the common risks of cloud data migration and how streaming data integration with low-impact change data capture minimizes both downtime and risks. Our second presenter, Edward Bell, gave us a live demonstration of Striim for zero downtime data migration. In this blog post, you can find my short recap of the key areas of the presentation. This summary certainly cannot do justice to the comprehensive discussion we had at the webinar. That’s why I highly recommend you watch the full webinar on-demand to access details on the solution architecture, its comparison to batch ETL approach, customer examples, the live demonstration, and the interactive Q&A section.

Cloud adoption brings multiple challenges and risks that prevent many businesses from modernizing their business-critical systems.

Limited cloud adoption and modernization reduces the ability to optimize business operations. These challenges and risks include causing downtime and business disruption and losing data during the migration, which are simply not acceptable for critical business systems. The risk list, however, is longer than these two. Switching over to cloud without adequate testing that leads to failures, working with stale data in the cloud, and data security and privacy are also among the key concerns.

Steve emphasized the point that “rushing the testing of the new environment to reduce the downtime, if you cannot continually feed data, can also lead to failures down the line or problems with the application.” Later, he added that “Beyond the migration, how do you continually feed the system? Especially in integration use cases where you are maintaining the data where it was and also delivering somewhere else, you need to continuously refresh the data to prevent staleness.”

Each of these risks mentioned above are preventable with the right approach to data movement between the legacy and new cloud systems.

 

Streaming data integration plays a critical role in successful cloud adoption with minimized risks.

A reliable, secure, and scalable streaming data integration architecture with low-impact change data capture enables zero database downtime and zero data loss during data migration. Because the source system is not interrupted, you can test the new cloud system as long as you need before the switchover. You also have the option to failback to the legacy system after switchover by reversing the data flow and keeping the old system up-to-date with the cloud system until you are fully confident that it is stable.

CDCInitialLoad.png” alt=”” width=”1902″ height=”958″ />

Striim’s cloud data migration solution uses this modern approach. During the bulk load, Striim’s CDC component collects the source database changes in real time. As soon as the initial load is complete, Striim applies the changes to the target environment to maintain the legacy and cloud database consistency. With built-in exactly once processing (E1P), Striim can avoid data both data loss and duplicates. You have the ability to use Striim’s real-time dashboards to monitor the data flow and various detailed performance metrics.

Continuous, streaming data integration for hybrid cloud architecture liberates your data for modernization and business transformation.

Cloud adoption and streaming integration are not limited to the lifting and shifting of your systems to the cloud. Ongoing integration post-migration is a crucial part of planning your cloud adoption. You cannot restrict it to database sources and database targets in the cloud, either. Your data lives in various systems and needs to be shared with different endpoints, such as your storage, data lake, or messaging systems in the cloud environment. Without enabling comprehensive and timely data flow from your enterprise systems to the cloud, what you can achieve in the cloud will be very limited.

“It is all about liberating your data.” Steve added in this part of the presentation. “Making it useful for the purpose you need it for. Continuous delivery in the correct format from a variety of sources relies on being able to filter that data, transform it, and possibly aggregate, join and enrich before you deliver to where needed. All of these can be done in Striim with a SQL-based language.”

A key point both Edward and Steve made is that Striim is very flexible. You can source from multiple sources and send to multiple targets. True data liberation and modernizing your data infrastructure needs that flexibility.

Striim also provides deployment flexibility. In fact, this was a question in the Q&A part, asking about deployment options and pricing. Unfortunately we could not answer all the questions we received. The short answer is: Striim can be deployed in the cloud, on-premises, or both via a hybrid topology. It is priced based on the CPUs of the servers where the Striim platform is installed. So you don’t need to worry about the sizes of your source and target systems.

There is much more covered in this short webinar we hosted on cloud adoption. I invite you to watch it on-demand at your convenience. If you would like to get a customized demo for cloud adoption or other streaming data integration use cases, please feel free to reach out.

Striim - 2019 CODiE Awards - Best iPaaS

Striim Is a 2019 CODiE Awards Finalist for Best iPaaS Solution

Striim is proud to announce that we’ve been recognized by SIIA as a 2019 CODiE Awards Finalist as a Best iPaaS, or Integration Platform as a Service.Striim - 2019 CODiE Awards - Best iPaaS

Why was Striim selected as a Best iPaaS solution? Striim is the only streaming (real-time) data integration platform running in the cloud that is built specifically to support cloud computing.

Real-time data integration is crucial for hybrid and multi-cloud architectures. Striim’s iPaaS solutions for real-time data integration in the cloud brings the agility and cost benefits of the cloud to integration use cases.

Striim enables companies to:

  • Quickly and easily provision streaming data pipelines to deliver real-time data to the cloud, or between cloud services
  • Easily adopt a multi-cloud architecture by seamlessly moving data across different cloud service providers: Azure, AWS, and Google Cloud
  • Offload operational workloads to cloud by moving data in real time and in the desired format
  • Filter, aggregate, transform, and enrich data-in-motion before delivering to the cloud in order to optimize cloud storage
  • Migrate data to the cloud without interrupting business operations
  • Minimize risk of cloud migrations with real-time, built-in cloud migration monitoring to avoid data divergence or data loss
  • Stream data in real time between cloud environments and back to on-premises systems

As one of the best iPaaS solutions, the Striim platform supports all aspects of Cloud integration as it relates to hybrid cloud and multi-cloud deployments.

Striim enables zero-downtime data migration to cloud by performing an initial load, and delivering the changes to the legacy system that occurred during the loading without pausing the source system. To prevent data loss, it validates that all of the data from on-premises sources migrated to the cloud environment.

Striim’s iPaaS solution provides the real-time data pipelines to and from the cloud to enable operational workloads in the cloud with the availability of up-to-date data.

Striim supports multi-cloud architecture by streaming data between different cloud platforms, including Azure, Google and AWS, and other cloud technologies such as Salesforce and Snowflake. If necessary, Striim can also provide real-time data flows between services offered within each of the three cloud platforms.

About Striim for Data IPaaS

Running as a PaaS solution on Microsoft Azure, AWS and Google Cloud Platform, the Striim streaming data integration platform offers real-time data ingestion from on-premises and cloud-based databases (including Oracle, SQL Server, HPE NonStop, PostgreSQL and MySQL), data warehouses (such as Oracle Exadata and Teradata), cloud services (such as AWS RDS and Amazon S3), Salesforce, log files, messaging systems (including Kafka), sensors, and Hadoop solutions.

Striim delivers this data in real time to a wide variety of cloud services (for example, Azure SQL Data Warehouse, Cosmos DB and Event Hubs; Amazon Redshift, S3 and Kinesis; and Google BigQuery, Cloud SQL and Pub/Sub), with in-flight transformations and enrichments.

Users can rapidly provision and deploy integration applications via a click-through interface using Striim’s pre-built templates and pre-configured integrations that are optimized for their cloud endpoints.

To learn more about Striim’s capabilities as one of the best iPaaS solutions, check out our three-part blog series, “Striim for Data iPaaS.”

What is iPaaS for Data?

What is iPaaS for Data?

Organizations can leverage a wide variety of cloud-based services today, and one of the fastest growing offerings is integration platform as a service. But what is iPaaS?

There are two major categories of iPaaS solutions available, focusing on application integration and data integration. Application integration works at the API level, typically involves relatively low volumes of messages, and enables multiple SaaS applications to be woven together.What is iPaaS for Data?

Integration platform as a service for data enables organizations to develop, execute, monitor, and govern integration across disparate data sources and targets, both on-premises and in the cloud, with processing and enrichment of the data as its streaming.

Within the scope of iPaaS for data there are older batch offerings, and more modern real-time streaming solutions. The latter are better suited to the on-demand and continuous way organizations are utilizing cloud resources.

Streaming data iPaaS solutions facilitate integration through intuitive UIs, by providing pre-configured connectors, automated operators, wizards and visualization tools to facilitate creation of data pipelines for real-time integration. With the iPaaS model, companies can develop and deploy the integrations they need without having to install or manage additional hardware or middleware, or acquire specific skills related to data integration. This can result in significant cost savings and accelerated deployment.

This is particularly useful as enterprise-scale cloud adoption becomes more prevalent, and organizations are required to integrate on-premises data and cloud data in real time to serve the company’s analytics and operational needs.

Factors such as increasing awareness of the benefits of iPaaS among enterprises – including reduced cost of ownership and operational optimization – are fueling the growth of the market worldwide.

For example, a report by Markets and Markets notes that the Integration Platform as a Service market is estimated to grow from $528 million in 2016 to nearly $3 billion by 2021, at a compound annual growth rate (CAGR) of 42% during the forecast period.

“The iPaaS market is booming as enterprises [embrace] hybrid and multi-cloud strategies to reduce cost and optimize workload performance” across on-premises and cloud infrastructure, the report says. Organizations around the world are adopting iPaaS and considering the deployment model an important enabler for their future, the study says.

Research firm Gartner, Inc. notes that the enterprise iPaaS market is an increasingly attractive space due to the need for users to integrate multi-cloud data and applications, with various on-premises assets. The firm expects the market to continue to achieve high growth rates over the next several years.

By 2021, enterprise iPaaS will be the largest market segment in application middleware, Gartner says, potentially consuming the traditional software delivery model along the way.

“iPaaS is a key building block for creating platforms that disrupt traditional integration markets, due to a faster time-to-value proposition,” Gartner states.

The Striim platform can be deployed on-premises, but is also available as an iPaaS solution on Microsoft Azure, Google Cloud Platform, and Amazon Web Services. This solution can integrate with on-premise data through a secure agent installation. For more information, we invite you to schedule a demo with one of our lead technologists, or download the Striim platform.

2019 Technology Predictions

19 For 19: Technology Predictions For 2019 and Beyond

Striim’s 2019 Technology Predictions article was originally published on Forbes.

With 2018 out the door, it’s important to take a look at where we’ve been over these past twelve months before we embrace the possibilities of what’s ahead this year. It has been a 2019 Technology Predictionsfast-moving year in enterprise technology. Modern data management has been a primary objective for most enterprise companies in 2018, evidenced by the dramatic increase in cloud adoption, strategic mergers and acquisitions and the rise of artificial intelligence (AI) and other emerging technologies.

Continuing on from my predictions for 2018, let’s take out the crystal ball and imagine what could be happening technology-wise in 2016.

2019 Technology Predictions for Cloud

• The center of gravity for enterprise data centers will shift faster towards cloud as enterprise companies continue to expand their reliance on the cloud for more critical, high-value workloads, especially for cloud-bursting and analytics applications.

• Technologies that enable real-time data distribution between different cloud and on-premises systems will become increasingly important for almost all cloud use-cases.

• With the acquisition of Red Hat, IBM may not directly challenge the top providers but will play an essential role through the use of Red Hat technologies across these clouds, private clouds and on-premise data centers in increasingly hybrid models.

• Portable applications and serverless computing will accelerate the move to multi-cloud and hybrid models utilizing containers, Kubernetes, cloud and multi-cloud management, with more and more automation provided by a growing number of startups and established players.

• As more open-source technologies mature in the big data and analytics space, they will be turned into scalable managed cloud services, cannibalizing the revenue of commercial companies built to support them.

2019 Technology Predictions for Big Data

• Despite consolidation in the big data space, as evidenced by the Cloudera/Hortonworks merger, enterprise investment in big data infrastructure will wane as more companies move to the cloud for storage and analytics. (Full disclosure: Cloudera is a partner of Striim.)

• As 5G begins to make its way to market, data will be generated at even faster speeds, requiring enterprise companies to seriously consider modernizing their architecture to work natively with streaming data and in-memory processing.

• Lambda and Kappa architectures combining streaming and batch processing and analytics will continue to grow in popularity driven by technologies that can work with both real-time and long-term storage sources and targets. Such mixed-use architectures will be essential in driving machine learning operationalization.

• Data processing components of streaming and batch big data analytics will widely adopt variants of the SQL language to enable self-service processing and analytics by users that best know the data, rather than developers that use APIs.

• As more organizations operate in real time, fast, scalable SQL-based architectures like Snowflake and Apache Kudu will become more popular than traditional big data environments, driven by the need for continual up-to-date information.

2019 Technology Predictions for Machine Learning/Artificial Intelligence

• AI and machine learning will no longer be considered a specialty and will permeate business on a deeper level. By adopting centralized cross-functional AI departments, organizations will be able to produce, share and reuse AI models and solutions to realize rapid return on investment (ROI).

• The biggest benefits of AI will be achieved through integration of machine learning models with other essential new technologies. The convergence of AI with internet of things (IoT), blockchain and cloud investments will provide the greatest synergies with ground-breaking results.

• Data scientists will become part of DevOps in order to achieve rapid machine learning operationalization. Instead of being handed raw data, data scientists will move upstream and work with IT specialists to determine how to source, process and model data. This will enable models to be quickly integrated with real-time data flows, as well as continually evaluating, testing and updating models to ensure efficacy.

2019 Technology Predictions for Security

• The nature of threats will shift from many small actors to larger stronger, possibly state-sponsored adversaries, with industrial rather than consumer data being the target. The sophistication of these attacks will require more comprehensive real-time threat detection integrated with AI to adapt to ever-changing approaches.

• As more organizations move to cloud analytics, security and regulatory requirements will drastically increase the need for in-flight masking, obfuscation and encryption technologies, especially around PII and other sensitive information.

2019 Technology Predictions for IoT

• IoT, especially sensors coupled with location data, will undergo extreme growth, but will not be purchased directly by major enterprises. Instead, device makers and supporting real-time processing technologies will be combined by integrators using edge processing and cloud-based systems to provide complete IoT-based solutions across multiple industries.

• The increased variety of IoT devices, gateways and supporting technologies will lead to standardization efforts around protocols, data collection, formatting, canonical models and security requirements.

2019 Technology Predictions for Blockchain

• The adoption of blockchain-based digital ledger technologies will become more widespread, driven by easy-to-operate and manage cloud offerings in Amazon Web Services (AWS) and Azure. This will provide enterprises a way to rapidly prototype supply chain and digital contract implementations. (Full disclosure: AWS and Azure are partners of Striim.)

• Innovative new secure algorithms, coupled with computing power advances, will speed up the processing time of digital ledger transactions from seconds to milliseconds or microseconds in the next few years, enabling high-velocity streaming applications to work with blockchain.

Whether or not any of these 2019 technology predictions come to pass, we can be sure this year will bring a mix of steady movement towards enterprise modernization, continued investment in cloud, streaming architecture and machine learning, and a smattering of unexpected twists and new innovations that will enable enterprises to think — and act — nimbly.

Any thoughts or feedback on my 2019 technology predictions? Please share on Steve’s LinkedIn page: https://www.linkedin.com/in/stevewilkes/  For more information on Striim’s solutions in the areas Cloud, Big Data, Security and IoT, please visit our Solutions page, or schedule a brief demo with one of our lead technologists.

Continuously Move Data to Snowflake

Enterprises must continuously move data to Snowflake to take full advantage of this data warehouse built for the cloud.

You chose Snowflake to provide rapid insights into your data on a massive scale, on AWS or Azure. However, most of your source data resides elsewhere – in a wide variety of on-premise or cloud sources. How do you continually move data to Snowflake in real-time, processing it along the way, so that your fast analytics and insights are reporting on timely data?

Snowflake was built for the cloud, and built for speed. By separating compute from storage you can easily scale up and down as needed. This gives you instant elasticity supporting any amount of data, and high speed queries for any number of users, coupled with the peace of mind provided by secure data sharing. The per-second pricing and support for multiple clouds allows you to choose your infrastructure and only pay when you are using the data warehouse.

However, residing in cloud means you have to determine how to most effectively move data to Snowflake. This could be migrating an existing Teradata or Exadata Data Warehouse, or continually populating Snowflake with newly generated on-premises data from operational databases, logs, or device information. In order for the warehouse to provide up-to-date information, there should be as little latency as possible between the original data creation and its delivery to Snowflake.

The Striim platform can help with all these requirements and more. Our database adapters support change data capture, or CDC, from enterprise or cloud databases. CDC directly intercepts database activity and collects all the inserts, updates, and deletes as they happen, ready to stream into Snowflake. Adapters for machine logs and other files read at the end of multiple files in parallel to stream out data as it is written, removing the inherent latency of batch. While data from devices and messaging systems can be collected easily, independent of their format, through a variety of high-speed adapters and parsers.

After being collected continuously, the streaming data can be delivered directly into Snowflake with very low latency, or pushed through a data pipeline where it can be pre-processed through filtering, transformation, enrichment, and correlation using SQL-based queries, before delivery into Snowflake. This enables such things as data denormalization, change detection, de-duplication, and quality checking before the data is ever stored.

In addition to this, because Striim is an enterprise grade platform, it can scale with Snowflake and reliably guarantee delivery of source data while also providing built-in dashboards and verification of data pipelines for operational monitoring purposes.

The Striim wizard-based UI enables users to rapidly create a new data flow to move data to Snowflake. In this example, real-time change data from Oracle is being continually delivered to Snowflake. The wizard walks you through all the configuration steps, checking that everything is set up properly, and results in a data flow application. This data flow can be enhanced to filter, transform and enrich the data through SQL-based queries. In the video, we add a name and email address from a cache, based on an ID present in the original data.

When the application is started, data flows in real-time from Oracle to Snowflake. Making changes in Oracle results in the transformed data being written continually to Snowflake, visible through the Snowflake UI.

Striim and Snowflake can change the way you do analytics, with Snowflake providing rapid insight to the real-time data provided by Striim. The data warehouse that is built for the cloud needs data delivered to the cloud, and Striim can continuously move data to Snowflake to support your business operations and decision-making.

To learn more about how Striim makes it easy to continuously move data to Snowflake, visit our Striim for Snowflake product page, schedule a demo with a Striim technologist, or download the platform and try it for yourself. 

Key Takeaways from AWS re:Invent 2018

Striim spent last week in Las Vegas exhibiting at AWS re:Invent, Amazon’s annual user conference for the global cloud computing community that featured keynote presentations, deep dive technical sessions, hackathons, and certification opportunities.AWS re:Invent

This was the biggest re:Invent ever, with 50,000+ cloud enthusiasts in attendance, and made for the perfect setting for some great conversations with potential customers, integrators and partners. Furthermore, we were able to really spend quality time talking to the community-at-large to more deeply understand their current data management practices, how they’re using their AWS environment, and what they’re looking for to further maximize their solutions.

Based on our week-long presence at the conference, below are some of our key takeaways from AWS re:Invent 2018:

  • Operationalizing ML Models: Amazon continues to double down on machine learning and artificial intelligence, and the name of the game is speed. Amazon announced the launch of the AWS Marketplace for ML and AI models, enabling developers to buy and sell machine learning models. Additionally, to help speed up training for AI models, Amazon announced multiple SageMaker offerings. While this will most certainly help build models, being able to build and deploy ML/AI models quickly is important, it’s also critical that the models aren’t being trained with high-latency data. This is why real-time streaming integration is a critical component in ML/AI initiatives.
  • Amazon Managed Streaming for Kafka (Amazon MSK): With the announcement of Amazon MSK, users can easily build and run applications that use Apache Kafka to process streaming data, which opens up numerous opportunities to interact with emerging technology. With Apache Kafka being one of the most popular open source platforms for building real-time data pipelines and applications, this managed service promotes what the industry has been seeing moving towards for the last few years – a streaming-first architecture.  
  • Amazon Builds Its Own Time Series DB – Amazon Timestream: Amazon created and launched its own time series DB, a fully managed database designed to track items over time. Geared towards IoT and operational applications, Amazon Timestream makes it easy to store and analyze trillions of events per day at a fraction of the cost of relational databases. This time series DB will be critical for helping organizations make real-time decisions based on measuring changes.
  • Blockchain Offerings: It was interesting to see that Amazon is still very much engaged in blockchain technology by providing ledger services powered by Ethereum and Hyperledger Fabric: Quantum Ledger Database and Amazon Managed Blockchain. According to AWS CEO Andy Jessy, this was in response to customer demand, so even though cryptocurrency is still in a volatile state, customers still see a tremendous amount of potential for this technology in other areas, and AWS is working to help them realize that vision.  

At the end of the day, data is a company’s most valuable asset and to get the most from it, adopting a cloud strategy offers numerous benefits. AWS re:Invent was a great event that showcased the future of cloud computing and how technology is enabling organizations to securely adopt a hybrid approach. We’re excited to see how the activities from the show translate over the next 12 months, especially the growing number of integration opportunities it presents, and can’t wait to be back next year.

To learn more about how the Striim platform can enhance your AWS solution, visit our “Striim for Amazon Web Services” product page, schedule a demo with a Striim technologist, or download a free trial of the platform.

 

Streaming Data Integration to AWS

As businesses adopt Amazon Web Services, streaming data integration to AWS – with change data capture (CDC) and stream processing – becomes a necessary part of the solution.

You’ve already decided that you want to enable integration to AWS. This could be to Amazon RDS or Aurora, Amazon Redshift, Amazon S3, Amazon Kinesis, Amazon EMR, or any number of other technologies.

You may want to migrate existing applications to AWS, scale elastically as necessary, or use the cloud for analytics or machine learning, but running applications in AWS, as VMs or containers, is only part of the problem. You also need to consider how to you move data to the cloud, ensure your applications or analytics are always up to date, and make sure the data is in the right format to be valuable.

The most important starting point is ensuring you can stream data to the cloud in real time. Batch data movement can cause unpredictable load on cloud targets, and has a high latency, meaning your data is often hours old. For modern applications, having up-to-the-second information is essential, for example to provide current customer information, accurate business reporting, or for real-time decision making.

integration to wasStreaming data integration to AWS from on-premise systems requires making use of appropriate data collection technologies. For databases, this is change data capture, or CDC, which directly and continuously intercepts database activity, and collects all the inserts, updates, and deletes as events, as they happen. Log data requires file tailing, which reads at the end of one or more files across potentially multiple machines and streams the latest records as they are written. Other sources like IoT data, or third party SaaS applications, also require specific treatment in order to ensure data can be streamed in real time.

Once you have streaming data, the next consideration is what processing is necessary to make the data valuable for your specific AWS destination, and this depends on the use-case.

For database migration or elastic scalability use-cases, where the target schema is similar to the source, moving raw data from on-premise databases to Amazon RDS or Aurora may be sufficient. The important consideration here is that the source applications typically cannot be stopped, and it takes time to do an initial load. This is why collecting and delivering database change, during and after the initial load, is essential for zero downtime migrations.

For real-time applications sourcing from Amazon Kinesis, or analytics use-cases built on Amazon Redshift or Amazon EMR, it may be necessary to perform stream processing before the data is delivered to the cloud. This processing can transform the data structure, and enrich it with additional context information, while the data is in-flight, adding value to the data and optimizing downstream analytics.

Striim’s streaming integration to AWS can continuously collect data from on-premise, or other cloud databases, and deliver to all of your Amazon Web Services endpoints. Striim can take care of initial loads, as well as CDC for the continuous application of change, and these data flows can be created rapidly, and monitored and validated continuously through our intuitive UI.

With Striim, your cloud migrations, scaling, and analytics can be built and iterated-on at the speed of your business, ensuring your data is always where you want it, when you want it.

To learn more about streaming integration to AWS with Striim, visit our “Striim for Amazon Web Services” product page, schedule a demo with a Striim technologist, or download a free trial of the platform.

Real-Time AWS Cloud Migration Monitoring: 3-Minute Demo

AWS cloud migration requires more than just being able to run in VMs or cloud containers. Applications rely on data, and that data needs to be migrated as well.

In most cases, the original applications are essential to the business, and cannot be stopped during this process. Since it takes time to migrate the data, and time to verify the application after migration, it is essential that data changes are collected, and delivered during and after that initial load.

As the data is so crucial to the business, and change data will be continually applied for a long time, mechanisms that verify that the data is delivered correctly are an important aspect of any AWS cloud migration.

In this Migration Monitoring Demo we will show how, by collecting change data from source and target and matching transactions applied to each in real time, you can ensure your cloud database is completely synchronized with on-premise, and detect any data divergence when migrating from an on-premise database.

AWS Cloud Migration Monitoring with Striim

The key challenges with monitoring AWS cloud migration include:

  • Enabling data migration without a production outage with monitoring during and after migration.
  • Detecting out-of-sync data should any divergence occur with this detection happening immediately at the time of divergence, preventing further data corruption.
  • Running the monitoring solution non-intrusively with low overhead and obtaining sufficient information to enable fast resynchronization

In our scenario, we are monitoring the migration of an on-premise application to AWS. A Striim dashboard shows real-time status, complete with alerts, and is powered by a continuously running data pipeline. The on-premise application uses an Oracle Database and cannot be stopped. The database transactions are continually replicated to an Amazon Aurora MySQL Database. The underlying migration solution could be either Striim’s Migration Solution or other solutions such as AWS DMS.  

The objective is to monitor ongoing migration of transactions and alert when any transactions go out-of-sync, indicating any potential data discrepancy. This is achieved in the Striim platform through its continuous query processing layer. Transactions are continuously collected from the source and target databases in real-time and matched within a time window. If matching transactions do not occur within a period of time, they are considered long-running. If no match occurs in an additional time period, the transaction is considered missing. Alerts are generated in both cases.

The number of alerts for missing transactions and long-running transactions are displayed in the dashboard. Transaction rates and operation activity are also available in the dashboard and can be displayed for all tables, or for critical tables and users.

You can immediately see live updates and alerts when the transactions do not get propagated to the target within a user configured window, with long-running transaction that eventually make it to the target also tracked.

The dashboard is user customizable making it easy to add additional visualizations for specific monitoring as necessary.

You have seen how Striim can be used for continuous monitoring of your on-premise to AWS cloud migration. For more information, visit our AWS solution page, schedule a demo with a Striim technologist, or get started immediately using a download from our website, or via the AWS marketplace.