Driving HOPTEK’s AI-powered system with streaming data pipelines

The Challenge

The main challenge for HOPTEK was to maintain real-time synchronization with the ever-changing status of trucks on the road. Any delays in data updates could cause discrepancies between real-world scenarios and the dispatch planning system. Furthermore, they needed a solution to put a manageable technological demand on their small team.

HOPTEK’s business model relies on aggregating their customer’s data from various sources including legacy systems such as Transportation Management Systems (TMS) and Electronic Logging Devices (ELD). They used a system called “completeness, accuracy, and timeliness” for data delivery but found that their previous vendor limited the speed at which the data flowed into HOPTEK. Determined to disrupt the status quo, HOPTEK decided to remain at the forefront of modernizing the trucking industry and pursue their search for a real-time ETL solution. 

Their customers’ needs primarily drove the decision to explore new providers for faster and more accurate dispatch solutions. They required an ETL system to process and deliver large volumes of data (10,000 – 100,000 records) in less than a minute.

The Solution: Striim Cloud – Fully managed Change Data Capture and Streaming SQL

After a rigorous evaluation process against competitors like AWS and Talend, HOPTEK chose Striim. The deciding factors were Striim’s superior speed, performance, user-friendly interface, and cost-effectiveness. In an industry where trucks are always moving in real time on the road providing a constant stream of data to the operators, speed is critical. Ayush Agrawal, Product Manager for HOPTEK elaborates that “Performance is critical because it allows trucking companies to reoptimize on the fly – the faster they can respond if a truck breaks down – the sooner that info gets to HOPTEK the sooner the trucking operator can reoptimize.”

Striim’s ability to reduce data ingestion and transformation time from five minutes to less than one minute were a critical factor for HOPTEK. Streaming SQL and in-stream transformations were the key components leveraged to optimize the performance of their pipelines. Moreover, Striim’s Out-Of-The-Box (OOTB) components and drag-and-drop interface enabled HOPTEK’s team to build data applications without heavy coding. 

Finally, Striim Cloud’s usage based pricing and the flexibility to strategically use credits for cost planning further cemented its appeal. 

Implementation

HOPTEK’s transition to Striim was smooth and streamlined, facilitated by Striim’s customer support and solutions team. After defining the scope with Striim’s team, HOPTEK started with the platform before shifting to the cloud.

The proof-of-concept exercises and the initial setup was completed within a matter of weeks, during which Striim’s comprehensive documentation and templates helped HOPTEK build applications faster. Within two weeks, HOPTEK was up and running with its first customer in a sandbox for testing. After that, they gradually moved other pipelines over to Striim, gaining confidence in the robustness and reliability of the Striim platform.

HOPTEK leveraged Striim’s Streaming SQL capabilities to perform aggregates with in-memory sliding windows that apply logic and case statements on real-time streaming data. HOPTEK also built their own UDFs (User Defined Functions) in Java to apply flexible procedural logic on data streams in the cases where SQL was not flexible enough.

HOPTEK successfully conducted a proof-of-concept (PoC) with Striim, demonstrating their capability to read CSV files from an AWS S3 bucket and efficiently write the data to a MySQL database in real time. The PoC primarily focused on processing and transforming customer, trucking, and fleet information using extensive SQL transformations, aggregations, and joins.

By leveraging Striim’s powerful data integration and streaming analytics platform, HOPTEK seamlessly ingested the CSV files from the AWS S3 bucket, ensuring a continuous flow of data from the source. Striim’s intuitive interface and robust connectors facilitated the integration process, enabling the extraction of valuable insights from the customer, trucking, and fleet data. *To enable real-time analytics, HOPTEK implemented advanced SQL transformations, aggregations, and joins using Striim’s SQL-based streaming analytics capabilities. These operations allowed them to perform complex data manipulations and derive meaningful patterns and correlations from the raw data streams.

Throughout the PoC, HOPTEK demonstrated their expertise in leveraging Striim’s platform to handle large volumes of data and process it in near real-time. By effectively utilizing Striim’s capabilities, HOPTEK efficiently executed the necessary SQL operations to enrich, cleanse, and standardize the customer, trucking, and fleet information.

The successful completion of this PoC highlights HOPTEK’s proficiency in leveraging Striim’s streaming data integration and analytics capabilities to empower organizations with real-time insights. With their demonstrated ability to process data from AWS S3, perform SQL transformations, aggregations, and joins, HOPTEK paves the way for organizations to unlock the full potential of their customer, trucking, and fleet information for enhanced decision-making and operational efficiency.

The Impact

Since implementing Striim, HOPTEK has seen transformative changes in its business operations. The significantly reduced data ingestion and transformation time led to quicker dispatch planning, allowing HOPTEK’s customers to see results up to 80% faster than their previous SLA.

Ayush Agrawal explains that “Now we have established this confidence so when we go to a new customer – we know they can build a data pipeline in less than a week – this confidence comes from the robustness and understanding we have established with Striim.”