Skip to main content

Understanding and managing backpressure

In Striim applications, the output of an "upstream" component is connected to the input of a "downstream" component by a stream. The two components may process data at different speeds: for example, a Database Reader source might be able to read data from an on-premise database faster than Database Writer can write it to a target database in the cloud.

The streams that connect these components can hold only a certain number of events. When a stream reaches its limit, it can not accept further events. This condition is known as backpressure. This inability to accept events from upstream components increases the risk of the upstream sources developing backpressure in turn.

Identifying backpressure

When a stream in an application is backpressured, Striim will send an alert (see Managing Smart Alerts).

Backpressured streams are rendered in red in the Striim web GUI, as in this example image:

A view from the Flow Designer that shows a link from a CsvDataSource source to a CsvToPosData target. The link displays a red circle with three stacked waves to indicate backpressure.

You can use the Latency report to identify the downstream component causing the backpressure. See Using the REPORT LATENCY command for details.

In the Tungsten console, use the mon <streamname> command to check the value of the Stream Full parameter. A stream in a backpressure condition has the value of Stream Full set to true.

A console screen showing the output of the 'mon' command. The command shows a table of properties and values. One of these pairs is Stream Full, with the value True.

Reducing backpressure

Backpressure reduction strategies depend on the particular structure of the flow in place and the nature of the sources, targets, other components, and streams connecting them. The event processing speed of the target can be increased by assigning more CPU resources. Alternately, the stream coming from the source can be divided among several identical targets working in parallel. See Creating multiple writer instances for details.Creating multiple writer instances

For some targets, adjusting the batch, flush, or upload policies can help manage backpressure. See File Writer for details.

Query code inefficiencies in a CQ stream can slow processing, leading to backpressure. Examine the code of backpressured CQ streams to find optimizations, such as dropping fields that are not used by downstream operations.

Changing the size of a window can sometimes reduce backpressure.