Cloud environments help businesses to modernize their existing systems and build next-generation systems to improve their business operations in an agile and cost-efficient way. Today, many organizations adopt a cloud-first strategy and use cloud infrastructure (IaaS) or platforms (PaaS) as their first choice when building new systems for their customers, partners, and employees.
Cloud platforms offer businesses many easy-to-use services that support various programming languages to build their applications with speed and flexibility. Especially, web-based or mobile customer-facing applications are easily built on cloud environments with fast time to market to gain competitive advantage.
Real-Time Data Distribution from Existing Systems with No Performance Degradation: Whether it is rehosting an existing business system or building a new system in a cloud platform, continuous data access from relevant data sources is a critical requirement. New or rehosted systems need to be a part of the data distribution architecture regardless of where they run – in-house or in the cloud. For many operational systems, data sharing needs to happen in real-time to enable smooth and reliable business process execution.
For example, if a cloud-based mobile application displays customers’ order data, orders information should be up-to-date in the cloud as soon as a new order comes into the ERP system. Otherwise, the delay in updating the cloud application can result in misinformation, confusion, and increased customer service calls. Such outcomes have a negative impact on the brand and increase operational costs for the business. For that reason, real-time data distribution from existing in-house or cloud-based systems to the cloud platform is a fundamental component of implementing a hybrid cloud architecture for business systems.
For a fully connected hybrid cloud architecture, data stored in databases, existing messaging systems, and other relevant operational sources, such as sensors and logs, should be all connected to cloud-based databases, messaging systems, and storage solutions that support the applications and services running in the cloud. Especially, continuously streaming data to the cloud-based messaging systems, such as Azure Event Hubs, Google Pub/Sub, and Amazon Kinesis help with distributing data to many other services offered by these cloud platforms, giving an expanded benefit for the effort. Once the real-time data pipelines to the cloud are set up, businesses can easily build new applications, and seamlessly adopt new cloud services to get the most operational value from the cloud environment.
A related consideration for data distribution from enterprise and other cloud databases to a new cloud environment is the impact on the source systems. While the above-mentioned CDC method is a great fit for this use case, it is important to use the non-intrusive, log-based CDC to minimize any impact on the source systems when they are in production use.
Ensuring Transactional Integrity and Reliability: When cloud databases are supporting the new or rehosted business applications, they need to reflect business transactions accurately. Otherwise, end-users may experience irreversible or highly-damaging operational errors. These errors can include not being able to process an order, account balance errors, providing inaccurate information to buyers, etc. With bulk data integration methods, transactional consistency cannot be ensured. It is not an automatic capability for CDC to ensure transactional integrity. Some streaming integration and logical replication solutions are designed to maintain the ACID properties of transactions (atomicity, consistency, isolation, and durability) when moving the data to cloud or other targets. If a change data capture solution is used for incremental data movement, it is critical to make sure that the CDC feature supports transactional integrity.
Avoiding any and all data loss and data duplicates, i.e. exactly once processing (E1P), is also critical to enable a consistent state during the data movement. Finding a solution that can recover properly from process interruptions and network outages is crucial to establish trust in the data in the cloud for business systems.
Compliance with Regulations: When planning to use cloud-based operational systems, organizations should also consider compliance with regulations for data privacy and security. With various across-region or state-specific laws in place, certain private customer data might not be allowed to be moved to a public cloud without proper data privacy and security assurances. A secure data integration solution that can mask and encrypt data in-flight, can provide the ability to stay compliant with privacy and security regulations.
Ability to Handle Large Data Volumes with Low Latency: The initial load-related considerations mentioned in the migration section also apply here for initiating new systems in the cloud. To get the application started, large volumes of data may need to be loaded to the cloud environment.
For both the initial load and ongoing integration needs, the data integration solution should be able to rapidly scale when data volumes increase, and maintain the low-latency data distribution requirement mentioned earlier.