The Wednesday Change Nobody Mentioned
Let me tell you a story on how a good dashboard goes bad. It rarely starts with the data team. More often, it starts with a reasonable one-line change somewhere else, on a Wednesday afternoon, that nobody thought to mention.
Picture an application team tidying up its ordering service. A developer renames a column from amount to order_total because the new name better reflects the business process. The change passes code review, clears every application test, and ships. From the application team’s point of view, it’s a solid improvement, and it is.
Overnight, the pipeline that replicates that table keeps running. The load either fails outright or, worse, quietly writes nulls where revenue used to be. By Thursday morning the revenue dashboard is wrong, an executive notices before the data team does, and the first question in the room is: “Why didn’t you catch this?”
Here’s the thing: the data team did nothing wrong. Their validation rules were sound, their ETL followed best practices, and all their reviews passed. The failure happened before any of that work began; in a system the data team doesn’t own and wasn’t told about.
That gap is what this post is about: why it exists, who owns it, and which Striim capabilities help your teams catch these issues before they reach your platforms and your revenue numbers.
Shift-Left Analytics Isn’t Really Analytics
Let me start by being honest about the name. It’s a misnomer.
“Shift-left” comes from software testing, where it means moving quality checks earlier in the lifecycle instead of finding defects at the end. Applied to data, it means catching problems as data is sourced and captured, and giving data teams visibility into the data and its structure before anything moves.
That’s quality, governance, and observability work, not analytics. Some vendors use the phrase to describe running analysis earlier, on streaming data for example, and that’s a legitimate idea. But the more useful reading focuses on the point of capture, and that’s the lens I’ll use for this post.
Your dashboards don’t change. What changes is where you look for trouble, and who’s accountable when you find it.
Warehouse Observability Is Too Late by Design
Observability in the warehouse detects problems when the damage has already been done—once the data has already moved and landed.
That isn’t a flaw in the tools. It’s simply about where they sit.
Software teams learned this lesson through testing: the later you find a defect, the more it costs to fix. Data works the same way. By the time a warehouse monitor fires, the bad records have been loaded, transformed, joined to other tables, and possibly used by someone to make a decision. Fixing that means tracing every downstream dependency, backfilling, and explaining the gap to stakeholders.
Loading raw data first and fixing it later, the ELT pattern, is often a sensible choice for many reasons. But by design, it places detection after movement. Warehouse monitors make you faster at finding damage. They don’t prevent it.
Moving left means watching the source as data is captured.
Schema Evolution: Stop the Change at the Point of Capture
Striim addresses this through its schema evolution capability. For supported change data capture (CDC) sources, schema evolution reads DDL changes from the source and lets you decide how each one is handled:
- Halt the application.
- Replicate the change to the target.
- Ignore the change.
- Quiesce the application.
Two DDL commands sit outside those choices. RENAME COLUMN and RENAME TABLE always halt the pipeline automatically.
Now apply that to Wednesday’s rename. Instead of a quiet null, the pipeline stops at the point of capture, before bad data reaches the warehouse. A halted pipeline is a loud, early failure. Paired with alerting, it tells the data team about the change when it happens, not after a dashboard breaks.
Continuous Queries: The Same Logic for Data in Flight
Continuous Queries (CQs) are SQL-like queries that run continuously on the pipeline to filter, aggregate, join, enrich, and transform events. A simple query can watch a stream for warning- and error-level events and raise an alert whenever one appears. A stream can also branch to several queries at once, so one path carries the data onward while another sends alerts by email, Slack, or Microsoft Teams.
Picture that on the order table. One query passes along only the records where the order total is present and valid. A second path raises an alert for the records that aren’t, and a separate target holds them for review. The bad records stay out of the warehouse, and the data team hears about them as they arrive.
One clarification so expectations are set correctly: this pattern is built from documented building blocks. It isn’t a packaged data-quality feature, so the rules are queries your team writes and maintains.
The principle holds whether a rule is a validation check or a masking step: the earlier it runs, the less there is to clean up.
Shift-left doesn’t replace warehouse observability. It changes its job, from discovering that something is wrong to confirming that everything is fine.
It’s an Org Chart Problem, Not a Tooling Problem
Data quality most often breaks where ownership is unclear. So the fix starts with who owns the source, not with which tool watches the warehouse.
Go back to my Wednesday example. The application team made a reasonable change inside its own system. The data team’s pipeline depended on that system, but nobody had agreed on what the table’s shape promised. Neither team was careless, but the organization simply never decided who owns that boundary.
That’s why the common advice, that data teams need to do a better job upstream, lands poorly. It’s also inaccurate. Data teams are guarding a door they don’t control. Adding more tests behind the door doesn’t change who decides what comes through it.
What does change the situation is making the expectation explicit and enforceable. That’s the idea behind a data contract: a shared agreement between the people who produce data and the people who depend on it. The same mechanism can serve very different agreements. An analytics team may want new tables added to its pipelines automatically, while a software development team may need every schema change blocked and flagged immediately. Same capability, different contract, set pipeline by pipeline.
At Striim, we built the halt action as the strict version of that contract. It fits tables where DDL changes should never occur at the source, because any change stops the pipeline until someone investigates. Tables with static schemas can run in their own application, reading from the same source stream, so a halt on one doesn’t stop data flowing for the rest.
Alerts are how the right people find out. Striim supports Smart Alerts, custom alerts, and application-generated alerts for data conditions, such as a field value exceeding a limit. Notifications can go to email, Slack, or Microsoft Teams, and rules can open a support ticket and assign it to the right team, a payments operations team for example. That’s routing by ownership, which is exactly the organizational point.
Here’s my take: none of this settles the organizational question on its own. A halt rule is the technical expression of a decision your organization still has to make: who owns the schema, who gets told about changes, and who can approve them. Tooling can enforce a contract. It can’t write one.
“Isn’t This Just Good ETL?”
Fair question. Good ETL discipline and shift-left protect different points in the data’s journey, and a mature team needs both.
ETL and ELT practices begin at the data team’s door. Extraction treats the source system as a given: you pull what’s there, validate it, transform it, and load it. Those practices are valuable, but they react to whatever the source emits, and ELT’s load-first pattern makes that more pronounced.
A common cause of broken pipelines is a source change that happens before any of that process begins. No amount of ETL discipline prevents it, because the change sits outside the process. Shift-left moves the boundary of responsibility upstream, toward making the source accountable for what it hands over.
The result is layered protection. Capture-time checks are the first layer, and validation after delivery is the backstop. They answer different questions:
- Capture-time checks ask whether the source changed what it promised.
- Post-delivery validation asks whether what arrived matches what left.
Validata: The Post-Delivery Backstop
Striim answers that second question with Validata, a separate product built for exactly that purpose. Validata compares source and target using full-dataset methods (Vector, Fast Record, Full Record), partial-dataset methods (Key, Interval), or custom checks you define. You can run a comparison once or schedule it hourly, daily, or weekly. When a pair comes back out of sync, Validata generates the SQL reconciliation scripts to fix it and alerts your team by email, Slack, or Microsoft Teams.
The working loop is straightforward:
- Compare source and target.
- Generate the repair.
- Re-verify that the fix held.
Back to my example. Suppose a change slipped past the capture-time checks, perhaps on a table the schema evolution settings didn’t cover. A scheduled comparison is the check that catches what those rules miss. When the target no longer matches the source, the pair comes back out of sync or halted, and Validata generates the reconciliation SQL. The gap surfaces on the next scheduled run, which can be as often as hourly, instead of when an executive spots a wrong number.
A word of caution: how a particular schema change shows up in a comparison depends on the change and the validation method, so test it against your own tables before you rely on it.
There’s a second audience for this, and it isn’t the engineers. Validata keeps its reports in a managed store, the Validata Historian, which gives leaders something rare in data work: a record of when a table was checked and what the result was. For teams in regulated industries, that history can matter as much as the check itself.
Freshness Is a Trust Signal
A table can be correct and still untrustworthy if nobody knows how old it is.
Schema changes aren’t the only way data goes wrong. Stale data looks fine: the dashboard loads, the numbers are plausible, and decisions get made on data that stopped updating hours ago. Catching that early is the same shift-left move, applied to time.
The measure to watch is end-to-end lag, the time from an event’s origin in the source to its final delivery at the target. Striim samples it every five seconds and keeps 24 hours of history, and you can read it in the web UI, the console, or the system health REST API. The shape of the number tells you where to look:
- High but stable lag usually means one of two things: the pipeline is reading stale data from the source, or something inside the pipeline, such as a window, is holding events.
- Steadily climbing lag means events are arriving faster than the target can absorb them.
Quiet sources are covered too. When a source goes inactive, Striim inserts idle events so lag keeps being tracked. Silence gets measured instead of being mistaken for health, and that’s the failure that tends to go unnoticed longest.
Freshness belongs in the contract as well. A freshness SLA works like a schema agreement: the data must arrive within an agreed window, and someone is alerted when it doesn’t. Striim’s Smart Alerts cover the common conditions out of the box, including application backpressure and an idle source.
Same principle as schema: you want to know the data is late when it’s late, not when a stakeholder asks why the numbers haven’t moved.
Honest Objections
Shift-left has real limits, and I’d rather name them here than have you discover them mid-rollout. Four objections come up most often.
“Upstream teams don’t want the extra work.” Fair, and it’s the objection that decides whether a program succeeds. The practical answer is to start small. Protect the handful of tables that feed the numbers leaders actually use, and keep the contract inside the workflow the application team already follows. Asking for less, and showing what it prevents, works better than asking for everything.
“Catching a change at capture is still catching it after the fact.” Also fair. Striim reads changes from the source’s own change stream, so by the time a halt fires, the producer has already made the change. What the halt protects is everything downstream. Stopping a breaking change before it ships takes a different mechanism, such as checks in the producer’s build, and that sits further left than capture-time observability.
“Coverage is uneven.” True, and you should know where before you commit. Schema evolution depends on the source and target pairing, and many pairings carry restrictions:
- The Snowflake and Databricks writers handle only some DDL operations.
- Modifying a column isn’t supported with BigQuery or Databricks targets.
- Schema evolution doesn’t work with bidirectional replication.
- It only sees changes made after it’s enabled.
- On SQL Server’s MSJet reader, schema changes made through SQL Server Management Studio are a known blind spot.
Validata has its own boundaries. It works across a defined set of databases and warehouses, and it doesn’t handle data types such as binary. The test is simple: map your actual sources and targets against these limits before you promise anyone full coverage.
“Central governance still matters.” It does. Capture-time checks cover structure and delivery, not meaning. Someone still has to decide what “revenue” means and make sure every team uses the same definition. Moving checks earlier can fragment those definitions if governance doesn’t keep pace.
Degrees of Left
Shift-left isn’t a switch. It’s a spectrum, and the useful question is how far left your organization can realistically move.
| Level | Where the check runs | What it catches | Who owns it |
|---|---|---|---|
| 1. Producer-side contract | In the application team’s build and release process | A breaking change before it ships | Application team |
| 2. Capture-time observability | At the source’s change stream, before data moves | Schema changes and stalled or stale feeds as they happen | Data team, with the application team informed |
| 3. Warehouse observability | After data lands | Bad or stale data that got through | Data team |
Level one is furthest left and the hardest to reach, because it requires agreement and tooling on the application side, such as data contracts, schema registries, and checks in the producer’s build. Level three is where warehouse monitoring tools operate, and it remains necessary. The Striim capabilities in this post work primarily at level two, with Validata serving as the backstop after delivery.
If your only checks live at level three, moving one level left is a realistic objective, and it addresses the Wednesday problem directly. Doing it well takes a decision about ownership as much as a purchase.
Let’s Execute on This
If you want to move one level left this quarter, here’s a practical path:
- Identify the handful of tables that feed the numbers your leaders use.
- Sit down with the owning application team and agree on what those tables promise, and who approves changes.
- Set schema evolution to halt for tables whose schemas should never change and run them in their own application.
- Route alerts to the team that owns the source, not only to the data team.
- Watch end-to-end lag and set a freshness expectation for each critical feed.
- Schedule Validata comparisons as your backstop, and test how your own schema changes surface before you depend on them.
The Bottom Line
Shift-left analytics isn’t analytics. At its core, it’s a decision about where trust gets established, and how early. The tooling helps, but the teams that get this right are the ones that decide together who owns the boundary.
So here’s my question for you: where does trust get established in your data today, and what would it take to move it one step left? I’d value hearing how your teams are tackling this. Share your experience in the comments or reach out directly. I’d be happy to compare notes.
Talk to Striim. Walk through your sources and targets with our team and see where capture-time checks fit in your stack. → Talk to Striim
Try Striim for free. Set up schema evolution and alerts on a pipeline of your own and see a halt fire before bad data lands. → Start your free trial
