Striim 3.10.1 Further Speeds Cloud Adoption

 

 

We are pleased to announce the general availability of Striim 3.10.1 that includes support for new and enhanced Cloud targets, extends manageability and diagnostics capabilities, and introduces new ease of use features to speed our customers’ cloud adoption. Key Features released in Striim 3.10.1 are directly available through Snowflake Partner Connect to enable rapid movement of enterprise data into Snowflake.

Striim 3.10.1 Focus Areas Including Cloud Adoption

This new release introduces many new features and capabilities, summarized here:

3.10.1 Features Summary

 

Let’s review the key themes and features of this new release, starting with the new and expanded cloud targets

Striim on Snowflake Partner Connect

From Snowflake Partner Connect, customers can launch a trial Striim Cloud instance directly as part of the Snowflake on-boarding process from the Snowflake UI and load data, optionally with change data capture, directly into Snowflake from any of our supported sources. You can read about this in a separate blog.

Expanded Support for Cloud Targets to Further Enhance Cloud Adoption

The Striim platform has been chosen as a standard for our customers’ cloud adoption use-cases partly because of the wide range of cloud targets it supports. Striim provides integration with databases, data warehouses, storage, messaging systems and other technologies across all three major cloud environments.

A major enhancement is the introduction of support for the Google BigQuery Streaming API. This not only enables real-time analytics on large scale data in BigQuery by ensuring that data is available within seconds of its creation, but it also helps with quota issues that can be faced by high volume customers. The integration through the BigQuery streaming API can support data transfer up to 1GB per second.

In addition to this, Striim 3.10.1 also has the following enhancements:

  • Optimized delivery to Snowflake and Azure Synapse that facilitates compacting multiple operations on the same data to a single operation on the target resulting in much lower change volume
  • Delivery to MongoDB cloud and MongoDB API for Azure Cosmos DB
  • Delivery to Apache Cassandra, DataStax Cassandra, and Cassandra API for Azure Cosmos DB

  • Support for delivery of data in Parquet format to Cloud Storage and Cloud Data Lakes to further support cloud analytics environments

Schema Conversion to Simplify Cloud Adoption Workflows

As part of many cloud migration or cloud integration use-cases, especially during the initial phases, developers often need to create target schemas to match those of source data. Striim adds the capability to use source schema information from popular databases such as Oracle, SQL Server, and PostgreSQL and create appropriate target schema in cloud targets such as Google BigQuery, Snowflake and others. Importantly, these conversions understand data type and structure differences between heterogeneous sources and targets and act intelligently to spot problems and inconsistencies before progressing to data movement, simplifying cloud adoption.

Enhanced Monitoring, Alerting and Diagnostics

On-going data movement between on-premise and cloud environments for migrations, or powering reporting and analytics solutions, are often part of an enterprise’s critical applications. As such they demand deep insights into the status of all active data flows.

Striim 3.10.1 adds the capability to inherently monitor data from its creation in the source to successful delivery in a target, generate detailed lag reports, and alert on situations where lag is outside of SLAs.

End to End Lag Visualization

In addition, this release provides detailed status on checkpointing information for recovery and high availability scenarios, with insight into checkpointing history and currency.

Real-time Checkpointing Information

Simplifies Working with Complex Data

As customers work with heterogeneous environments and adopt more complex integration scenarios, they often have to work with complex data types, or perform necessary data conversions. While always possible through user defined functions, this release adds multiple commonly requested data manipulation functions out of the box. This simplifies working with JSON data and document structures, while also facilitating data cleansing, and regular expression operations.

On-Going Support for Enterprise Sources

As customers upgrade their environments, or adopt new technologies, it is essential that their integration platform keeps pace. In Striim 3.10.1 we extend our support for the Oracle database to include Oracle 19c, including change data capture, add support for schema information and metadata for Oracle GoldenGate trails, and certify our support for Hive 3.1.0

These are a high level view of the new features of Striim 3.10.1. There is a lot more to discover to aid on your cloud adoption journey. If you would like to learn more about the new release, please reach out to schedule a demo with a Striim expert.

Cloud Adoption: How Streaming Integration Minimizes Risks

 

 

Last week, we hosted a live webinar, Cloud Adoption: How Streaming Integration Minimizes Risks. In just 35 minutes, we discussed how to eliminate database downtime and minimize other risks of cloud migration and ongoing integration for hybrid cloud architecture, including a live demo of Striim’s solution.

Our first speaker, Steve Wilkes, started the presentation discussing the importance of cloud adoption for today’s pandemic-impacted, fragile business environment. He continued with the common risks of cloud data migration and how streaming data integration with low-impact change data capture minimizes both downtime and risks. Our second presenter, Edward Bell, gave us a live demonstration of Striim for zero downtime data migration. In this blog post, you can find my short recap of the key areas of the presentation. This summary certainly cannot do justice to the comprehensive discussion we had at the webinar. That’s why I highly recommend you watch the full webinar on-demand to access details on the solution architecture, its comparison to batch ETL approach, customer examples, the live demonstration, and the interactive Q&A section.

Cloud adoption brings multiple challenges and risks that prevent many businesses from modernizing their business-critical systems.

Limited cloud adoption and modernization reduces the ability to optimize business operations. These challenges and risks include causing downtime and business disruption and losing data during the migration, which are simply not acceptable for critical business systems. The risk list, however, is longer than these two. Switching over to cloud without adequate testing that leads to failures, working with stale data in the cloud, and data security and privacy are also among the key concerns.

Steve emphasized the point that “rushing the testing of the new environment to reduce the downtime, if you cannot continually feed data, can also lead to failures down the line or problems with the application.” Later, he added that “Beyond the migration, how do you continually feed the system? Especially in integration use cases where you are maintaining the data where it was and also delivering somewhere else, you need to continuously refresh the data to prevent staleness.”

Each of these risks mentioned above are preventable with the right approach to data movement between the legacy and new cloud systems.

 

Streaming data integration plays a critical role in successful cloud adoption with minimized risks.

A reliable, secure, and scalable streaming data integration architecture with low-impact change data capture enables zero database downtime and zero data loss during data migration. Because the source system is not interrupted, you can test the new cloud system as long as you need before the switchover. You also have the option to failback to the legacy system after switchover by reversing the data flow and keeping the old system up-to-date with the cloud system until you are fully confident that it is stable.

CDCInitialLoad.png” alt=”” width=”1902″ height=”958″ />

Striim’s cloud data migration solution uses this modern approach. During the bulk load, Striim’s CDC component collects the source database changes in real time. As soon as the initial load is complete, Striim applies the changes to the target environment to maintain the legacy and cloud database consistency. With built-in exactly once processing (E1P), Striim can avoid data both data loss and duplicates. You have the ability to use Striim’s real-time dashboards to monitor the data flow and various detailed performance metrics.

Continuous, streaming data integration for hybrid cloud architecture liberates your data for modernization and business transformation.

Cloud adoption and streaming integration are not limited to the lifting and shifting of your systems to the cloud. Ongoing integration post-migration is a crucial part of planning your cloud adoption. You cannot restrict it to database sources and database targets in the cloud, either. Your data lives in various systems and needs to be shared with different endpoints, such as your storage, data lake, or messaging systems in the cloud environment. Without enabling comprehensive and timely data flow from your enterprise systems to the cloud, what you can achieve in the cloud will be very limited.

“It is all about liberating your data.” Steve added in this part of the presentation. “Making it useful for the purpose you need it for. Continuous delivery in the correct format from a variety of sources relies on being able to filter that data, transform it, and possibly aggregate, join and enrich before you deliver to where needed. All of these can be done in Striim with a SQL-based language.”

A key point both Edward and Steve made is that Striim is very flexible. You can source from multiple sources and send to multiple targets. True data liberation and modernizing your data infrastructure needs that flexibility.

Striim also provides deployment flexibility. In fact, this was a question in the Q&A part, asking about deployment options and pricing. Unfortunately we could not answer all the questions we received. The short answer is: Striim can be deployed in the cloud, on-premises, or both via a hybrid topology. It is priced based on the CPUs of the servers where the Striim platform is installed. So you don’t need to worry about the sizes of your source and target systems.

There is much more covered in this short webinar we hosted on cloud adoption. I invite you to watch it on-demand at your convenience. If you would like to get a customized demo for cloud adoption or other streaming data integration use cases, please feel free to reach out.

What is iPaaS for Data?

What is iPaaS for Data?

Organizations can leverage a wide variety of cloud-based services today, and one of the fastest growing offerings is integration platform as a service. But what is iPaaS?

There are two major categories of iPaaS solutions available, focusing on application integration and data integration. Application integration works at the API level, typically involves relatively low volumes of messages, and enables multiple SaaS applications to be woven together.What is iPaaS for Data?

Integration platform as a service for data enables organizations to develop, execute, monitor, and govern integration across disparate data sources and targets, both on-premises and in the cloud, with processing and enrichment of the data as its streaming.

Within the scope of iPaaS for data there are older batch offerings, and more modern real-time streaming solutions. The latter are better suited to the on-demand and continuous way organizations are utilizing cloud resources.

Streaming data iPaaS solutions facilitate integration through intuitive UIs, by providing pre-configured connectors, automated operators, wizards and visualization tools to facilitate creation of data pipelines for real-time integration. With the iPaaS model, companies can develop and deploy the integrations they need without having to install or manage additional hardware or middleware, or acquire specific skills related to data integration. This can result in significant cost savings and accelerated deployment.

This is particularly useful as enterprise-scale cloud adoption becomes more prevalent, and organizations are required to integrate on-premises data and cloud data in real time to serve the company’s analytics and operational needs.

Factors such as increasing awareness of the benefits of iPaaS among enterprises – including reduced cost of ownership and operational optimization – are fueling the growth of the market worldwide.

For example, a report by Markets and Markets notes that the Integration Platform as a Service market is estimated to grow from $528 million in 2016 to nearly $3 billion by 2021, at a compound annual growth rate (CAGR) of 42% during the forecast period.

“The iPaaS market is booming as enterprises [embrace] hybrid and multi-cloud strategies to reduce cost and optimize workload performance” across on-premises and cloud infrastructure, the report says. Organizations around the world are adopting iPaaS and considering the deployment model an important enabler for their future, the study says.

Research firm Gartner, Inc. notes that the enterprise iPaaS market is an increasingly attractive space due to the need for users to integrate multi-cloud data and applications, with various on-premises assets. The firm expects the market to continue to achieve high growth rates over the next several years.

By 2021, enterprise iPaaS will be the largest market segment in application middleware, Gartner says, potentially consuming the traditional software delivery model along the way.

“iPaaS is a key building block for creating platforms that disrupt traditional integration markets, due to a faster time-to-value proposition,” Gartner states.

The Striim platform can be deployed on-premises, but is also available as an iPaaS solution on Microsoft Azure, Google Cloud Platform, and Amazon Web Services. This solution can integrate with on-premise data through a secure agent installation. For more information, we invite you to schedule a demo with one of our lead technologists, or download the Striim platform.

2019 Technology Predictions

19 For 19: Technology Predictions For 2019 and Beyond

Striim’s 2019 Technology Predictions article was originally published on Forbes.

With 2018 out the door, it’s important to take a look at where we’ve been over these past twelve months before we embrace the possibilities of what’s ahead this year. It has been a 2019 Technology Predictionsfast-moving year in enterprise technology. Modern data management has been a primary objective for most enterprise companies in 2018, evidenced by the dramatic increase in cloud adoption, strategic mergers and acquisitions and the rise of artificial intelligence (AI) and other emerging technologies.

Continuing on from my predictions for 2018, let’s take out the crystal ball and imagine what could be happening technology-wise in 2016.

2019 Technology Predictions for Cloud

• The center of gravity for enterprise data centers will shift faster towards cloud as enterprise companies continue to expand their reliance on the cloud for more critical, high-value workloads, especially for cloud-bursting and analytics applications.

• Technologies that enable real-time data distribution between different cloud and on-premises systems will become increasingly important for almost all cloud use-cases.

• With the acquisition of Red Hat, IBM may not directly challenge the top providers but will play an essential role through the use of Red Hat technologies across these clouds, private clouds and on-premise data centers in increasingly hybrid models.

• Portable applications and serverless computing will accelerate the move to multi-cloud and hybrid models utilizing containers, Kubernetes, cloud and multi-cloud management, with more and more automation provided by a growing number of startups and established players.

• As more open-source technologies mature in the big data and analytics space, they will be turned into scalable managed cloud services, cannibalizing the revenue of commercial companies built to support them.

2019 Technology Predictions for Big Data

• Despite consolidation in the big data space, as evidenced by the Cloudera/Hortonworks merger, enterprise investment in big data infrastructure will wane as more companies move to the cloud for storage and analytics. (Full disclosure: Cloudera is a partner of Striim.)

• As 5G begins to make its way to market, data will be generated at even faster speeds, requiring enterprise companies to seriously consider modernizing their architecture to work natively with streaming data and in-memory processing.

• Lambda and Kappa architectures combining streaming and batch processing and analytics will continue to grow in popularity driven by technologies that can work with both real-time and long-term storage sources and targets. Such mixed-use architectures will be essential in driving machine learning operationalization.

• Data processing components of streaming and batch big data analytics will widely adopt variants of the SQL language to enable self-service processing and analytics by users that best know the data, rather than developers that use APIs.

• As more organizations operate in real time, fast, scalable SQL-based architectures like Snowflake and Apache Kudu will become more popular than traditional big data environments, driven by the need for continual up-to-date information.

2019 Technology Predictions for Machine Learning/Artificial Intelligence

• AI and machine learning will no longer be considered a specialty and will permeate business on a deeper level. By adopting centralized cross-functional AI departments, organizations will be able to produce, share and reuse AI models and solutions to realize rapid return on investment (ROI).

• The biggest benefits of AI will be achieved through integration of machine learning models with other essential new technologies. The convergence of AI with internet of things (IoT), blockchain and cloud investments will provide the greatest synergies with ground-breaking results.

• Data scientists will become part of DevOps in order to achieve rapid machine learning operationalization. Instead of being handed raw data, data scientists will move upstream and work with IT specialists to determine how to source, process and model data. This will enable models to be quickly integrated with real-time data flows, as well as continually evaluating, testing and updating models to ensure efficacy.

2019 Technology Predictions for Security

• The nature of threats will shift from many small actors to larger stronger, possibly state-sponsored adversaries, with industrial rather than consumer data being the target. The sophistication of these attacks will require more comprehensive real-time threat detection integrated with AI to adapt to ever-changing approaches.

• As more organizations move to cloud analytics, security and regulatory requirements will drastically increase the need for in-flight masking, obfuscation and encryption technologies, especially around PII and other sensitive information.

2019 Technology Predictions for IoT

• IoT, especially sensors coupled with location data, will undergo extreme growth, but will not be purchased directly by major enterprises. Instead, device makers and supporting real-time processing technologies will be combined by integrators using edge processing and cloud-based systems to provide complete IoT-based solutions across multiple industries.

• The increased variety of IoT devices, gateways and supporting technologies will lead to standardization efforts around protocols, data collection, formatting, canonical models and security requirements.

2019 Technology Predictions for Blockchain

• The adoption of blockchain-based digital ledger technologies will become more widespread, driven by easy-to-operate and manage cloud offerings in Amazon Web Services (AWS) and Azure. This will provide enterprises a way to rapidly prototype supply chain and digital contract implementations. (Full disclosure: AWS and Azure are partners of Striim.)

• Innovative new secure algorithms, coupled with computing power advances, will speed up the processing time of digital ledger transactions from seconds to milliseconds or microseconds in the next few years, enabling high-velocity streaming applications to work with blockchain.

Whether or not any of these 2019 technology predictions come to pass, we can be sure this year will bring a mix of steady movement towards enterprise modernization, continued investment in cloud, streaming architecture and machine learning, and a smattering of unexpected twists and new innovations that will enable enterprises to think — and act — nimbly.

Any thoughts or feedback on my 2019 technology predictions? Please share on Steve’s LinkedIn page: https://www.linkedin.com/in/stevewilkes/  For more information on Striim’s solutions in the areas Cloud, Big Data, Security and IoT, please visit our Solutions page, or schedule a brief demo with one of our lead technologists.

Move Data to Amazon Redshift with Striim

Continuously Move Data to Amazon Redshift via AWS Marketplace

Striim’s New Metered Cloud Solution for Streaming Data Pipelines to Move Data to Amazon Redshift Now Available in the AWS Marketplace

 

We are delighted to announce that Striim for Amazon Redshift is now available as a Platform-as-a-Service (PaaS) offering in the Amazon Web Services (AWS) Marketplace to enable companies to migrate and continuously move data to Amazon Redshift in real time. As an AWS Partner Network partner, we make it fast and easy to build streaming data pipelines to move data from a broad range of data sources to Amazon Redshift, speeding adoption of a hybrid-cloud architecture running on AWS.Move Data to Amazon Redshift with Striim

Running on AWS as a PaaS solution, the Striim platform offers non-intrusive, real-time data collection and movement from databases (including Oracle, SQL Server, HPE NonStop, PostgreSQL, and MySQL), data warehouses (such as Oracle Exadata and Teradata), Salesforce, Amazon S3, log files, messaging systems, sensors, and Hadoop solutions.

While data is streaming, Striim provides in-flight transformations and optimized delivery to Amazon Redshift.

“With Striim, AWS users can move data to Amazon Redshift continuously, and in the right format. Now that Striim for Amazon Redshift is available in the AWS Marketplace, streaming data pipelines to Redshift can be built in minutes using Striim’s data movement wizards. More importantly, Striim supports mission-critical workloads in the most demanding data environments, handling extreme volumes of data with built-in security and reliability for enterprise-grade, operational decision making.”

Alok Pareek
Founder and EVP of Products, Striim

For anyone looking to move data to Amazon Redshift, Striim offers several features and benefits that can maximize the speed and reliability of data migration, continuous data movement, and in-stream processing. Striim:

  • minimizes impact on source databases with non-intrusive change data capture (CDC)
  • simplifies CDC configuration through wizards
  • enables in-flight transformations / visualizations before delivery to Amazon Redshift
  • reduces data latency and on-premises ETL workloads
  • offers optimized interfaces to enable fast data loading to Amazon Redshift
  • provides full context for downstream operations

Amazon Redshift is a fast, scalable data warehouse that makes it simple and cost-effective to analyze all your data across your data warehouse and data lake. Redshift uses machine learning, massively parallel query execution, and columnar storage on high-performance disk to deliver high performance for cloud analytics.

For more information about Striim’s platform-as-a-service offering to move data to Amazon Redshift, please visit https://www.striim.com/partners/striim-for-aws/, or provision Striim for Amazon Redshift in the AWS Marketplace.