Skip to main content

ALTER and RECOMPILE

Use these commands to modify applications that are loaded but not running (in other words, applications with status Created or Undeployed). ALTER reopens the app for editing so that new components are added to that app. RECOMPILE verifies the app's validity and closes it. Altering apps is necessary in many situations such as when there is a DDL change to a source database table.

ALTER and RECOMPILE workflow

  1. If the application is running, STOP or QUIESCE it (see Console commands).

  2. If the application is deployed, undeploy it.

  3. Alter the application using CREATE, DROP, or CREATE OR REPLACE statements, as described in the example below.

  4. Recompile, deploy, and start the application.

How ALTER affects data

Altering an app can affect how it creates data and how it interacts with data already written to disk. Many changes can be made safely while others may make newly created data different from previously created data, causing errors. For example:

Generally safe changes

  • Add components or flows: adding a component may introduce new output data, such as adding a new target, but other components which interact with outside data would not be affected.

  • Add a field to a type: new components can use new fields on a Striim type while unaltered components will be unaffected.

  • Expand a data type: changing a smaller data type to a larger one typically means the larger one can handle all of the smaller values. For example, changing a byte to an integer is safe because integer can hold any value that byte can hold. Take care, however, that external data systems can receive and handle the larger types. Such generally safe changes include changing a type from:

    • byte to integer, short, or long

    • integer to short or long

    • short to long

    • float to double

Potentially unsafe changes

  • Removing or altering components: removing a target or WactionStore can obviously mean data not being written, but also removing a component in the middle of a flow can break data continuity. Or, altering a component in the middle of a flow can change how, when, or whether data is written by a latter component. Take care to maintain the integrity of data formats and the coherence of data flows.

  • Remove a field from a type: data processing and persistence may be affected by the removal of a field form a type. For example, a CQ may rely on it. Take care not to remove data fields which are necessary for the app to produce valid output.

  • Contract a data type: changing a larger data type to a smaller one can mean that the smaller type cannot hold all the values of the larger type. For example, changing a double to a float mean losing precision which can affect CQ processing, value matches, etc. Such potentially unsafe changes include changing a type from:

    • long to short, integer, or byte

    • short to integer or byte

    • integer to byte

    • double to float

How ALTER affects recovery

When you alter an app it retains its recovery checkpoint and will attempt to start with no missing or duplicate events (see Recovering applications) but some changes can cause incompatibilities.

Generally safe changes

  • Adding a branch to a data flow.

  • Removing a branch from a data flow.

  • Simple modifications to a CQ.

Potentially unsafe changes

  • Modifying sources. For example, changing the Wildcard (file name) property in File Reader would prevent recovery because the recovery checkpoint would have the previous file name.

  • Changing a window's KEEP clause. Changing the size of window batches can cause data to contain gaps or duplicates. See "Using QUIESCE before ALTER" below.

  • Changing a CQ's GROUP BY, FROM, LIMIT, or MATCH_PATTERN selection. Changing how CQs work on data can cause gaps or duplicates. See "Using QUIESCE before ALTER" below.

  • Changing a KafkaWriter's mode from sync to async.

Using QUIESCE before ALTER

Using QUIESCE before ALTER can mitigate some problems when making potentially unsafe changes to apps. If alterations change how the data streams are batched or routed, then when the app restarts the output could contain gaps or duplicates. If the app is first flushed using QUIESCE then it will be in a state where previous data batches do not affect recovery, so changes to window sizes or CQ processing parameters can take effect without undermining exactly once processing. see QUIESCE for details of how that command affects data output.

ALTER and RECOMPILE examples

To begin altering an application, use:

USE <namespace containing application>;
ALTER APPLICATION <application name>;

At this point, enter CREATE, DROP, or CREATE OR REPLACE statements to modify the application, then complete the alteration with:

ALTER APPLICATION <application name> RECOMPILE;
Example #1: adding an alert subscription

For example, to add the email subscription described in Sending alerts from applications to the PosApp sample application:

USE Samples;
ALTER APPLICATION PosApp;
CREATE SUBSCRIPTION PosAppEmailAlert
USING EmailAdapter (
  SMTPUSER:'sender@example.com',
  SMTPPASSWORD:'password', 
  smtpurl:'smtp.gmail.com',
  starttls_enable:'true',
  subject:"test subject",
  emailList:"recipient@example.com,recipient2.example.com",
  senderEmail:"alertsender@example.com" 
)
INPUT FROM AlertStream;
ALTER APPLICATION PosApp RECOMPILE;

At this point you may deploy and start the modified application. If recovery was enabled for the application when it was loaded, when it is restarted, it will pick up source data back to the time the app went offline (subject to the usual limitations detailed in Recovering applications).

Example #2: removing a field from a type

Keep in mind that a change made to one component may require changes to multiple downstream components and their types. For example, to add the event_url JSON source property to the following app you would need to modify both the MeetupJSONType and the ParseJSON CQ:

CREATE TYPE MeetupJSONType (
  venue_id integer KEY,
  group_name string,
  event_name string,
  event_url string,
  time DateTime,
  venue_name string,
  group_city string,
  group_country string,
  lat double,
  lon double
);
CREATE STREAM ParsedJSONStream OF MeetupJSONType;

CREATE CQ ParseJSON
INSERT INTO ParsedJSONStream
SELECT
  CASE
    WHEN data.has("venue") and data.get("venue").has("venue_id")
      THEN data.get("venue").get("venue_id").intValue()
    ELSE 0
  END,
  CASE
    WHEN data.has("group") and data.get("group").has("group_name")
      THEN data.get("group").get("group_name").textValue()
    ELSE "NA"
  END,
  CASE
    WHEN data.has("event") and data.get("event").has("event_name")
      THEN data.get("event").get("event_name").textValue()
    ELSE "NA"
  END,
  CASE
    WHEN data.has("event") and data.get("event").has("event_url")
      THEN data.get("event").get("event_url").textValue()
    ELSE "NA"
  END, ...

If the application has a dashboard, you might also need to edit its properties to add event_url there:

alterMeetup.png