Data quality checks are the tripwires of a production pipeline. They’re the assertions that tell you a schema drifted, a batch arrived late, a stream stopped emitting, or a column that should be 99.9% non-null suddenly became 40% null. In operational data engineering, these checks sit alongside schema evolution, failure recovery, and the maintenance burden of data infrastructure as one of the four forces that determine whether your pipelines are a source of operational advantage or a source of 3 a.m. pages. The problem isn’t that teams lack checks. The problem is that most teams build checks that fire too often, too vaguely, or too late, and then they train themselves to ignore the very signals that were supposed to protect them.
This article is for mid-career practitioners who run batch and streaming pipelines in production. You already know how to write a dbt test, a Great Expectations expectation, or a custom SQL assertion. What you may not have fully internalized is how to design the alerting layer around those checks so that every page, Slack message, or PagerDuty incident is worth the cognitive load it creates. Alert fatigue is not a people problem. It’s a design problem. And it’s solvable, but only if you’re willing to treat your checks as a system with its own failure modes, maintenance costs, and tradeoffs.
What Alert Fatigue Actually Costs You
Alert fatigue is the gradual erosion of attention that happens when a monitoring system produces more signals than a human can meaningfully act on. In clinical settings, researchers have documented that high rates of false or low-priority alarms lead nurses and physicians to delay responses, override alarms, or disable them entirely. The same pattern shows up in data engineering. When a pipeline emits 40 warnings a day and only two of them represent real data corruption, the team learns to treat all 40 as noise. The two real incidents get buried. The on-call rotation becomes a ritual of acknowledgment rather than investigation.
The cost isn’t just missed incidents. It’s the slow death of trust in the data itself. Downstream consumers—analysts, product managers, finance teams—start to assume that the warehouse is “always broken” or “always fine,” depending on which alerts they happen to see. Neither assumption is useful. The operational data engineer’s job is to make the state of the data legible, not to flood the channel with undifferentiated noise.
Separate Data Quality from Pipeline Health
The first design mistake is conflating two different questions: “Did the pipeline run?” and “Is the data correct?” These are related but not identical. A pipeline can run successfully and produce garbage. A pipeline can fail and leave the previous day’s data perfectly intact. If you route both types of events to the same alerting channel with the same severity, you force your team to triage every message manually.
Instead, create two distinct alert classes:
- Pipeline health alerts: job failures, retries, timeouts, resource exhaustion, late arrivals. These are operational signals about the machinery.
- Data quality alerts: schema drift, null-rate violations, distribution shifts, referential integrity breaks, duplicate keys. These are signals about the content.
Pipeline health alerts should go to the on-call rotation with clear runbook links. Data quality alerts should go to a separate channel—ideally a dedicated Slack room or a dashboard—where they can be reviewed during working hours unless they meet a high-severity threshold. The threshold is the key. A null rate that jumps from 0.1% to 0.5% is a data quality issue worth investigating, but it’s not a page-at-2 a.m. issue. A null rate that jumps from 0.1% to 40% on a column that feeds a revenue report is a page-at-2 a.m. issue. The difference isn’t the check itself; it’s the severity classification you attach to the check’s output.
Define Severity Before You Define Thresholds
Most teams start with thresholds and then try to reverse-engineer severity from the number of alerts that fire. That’s backwards. Start with the business impact of a data quality failure, then work down to the threshold that would trigger that impact.
For each critical table or stream, ask three questions:
- Who consumes this data, and what decision or process depends on it?
- What is the smallest deviation from expected values that would change that decision or process?
- How quickly does that deviation need to be caught before the cost becomes unacceptable?
The answers give you a severity ladder. A table that feeds a daily executive dashboard might have a 24-hour detection window and a 5% tolerance for row-count drift. A table that feeds a real-time fraud model might have a 5-minute detection window and a 0.1% tolerance for schema changes. The thresholds aren’t arbitrary numbers; they’re derived from the operational contract you have with downstream consumers.
This is where most data quality frameworks fall short. They give you a library of checks—null checks, uniqueness checks, range checks, freshness checks—but they don’t give you a method for deciding which checks deserve a page, which deserve a Slack message, and which deserve to be silently logged for weekly review. That method has to come from your team’s understanding of the business, not from the tool.
Design Checks for Actionability, Not Coverage
A common anti-pattern is the “checklist” approach: run 200 generic checks on every table, then tune the alerting to suppress the noise. This creates a maintenance burden that grows linearly with the number of tables, and it guarantees that the checks themselves become stale. A check that hasn’t fired in six months isn’t a sign of health; it’s a sign that the check is no longer aligned with the data’s actual failure modes.
Instead, design checks around the specific failure modes you’ve actually observed or can reasonably predict. If you run a batch pipeline that ingests third-party CSV files, you know that the most common failures are: missing files, malformed rows, encoding changes, and column reordering. Write checks for those four failure modes. Don’t write a check for “all values in the customer_id column are positive integers” unless you have a reason to believe that negative or non-integer values are a realistic failure mode. Every check you add is a liability: it costs compute time, it costs maintenance time, and it costs attention when it fires.
The same principle applies to streaming pipelines. If you run a Kafka-to-warehouse pipeline, the failure modes you care about are: consumer lag, schema registry mismatches, poison messages, and silent drops. Write checks for those. Don’t write a check for “average message size is within 2 standard deviations of the 30-day mean” unless you have evidence that message size anomalies correlate with data corruption. Unvalidated checks are just noise generators with extra steps.
Use Anomaly Detection Sparingly and Skeptically
Anomaly detection is often sold as the solution to alert fatigue: instead of setting static thresholds, let the system learn what “normal” looks like and alert on deviations. In practice, anomaly detection on data quality metrics creates a new kind of fatigue: the fatigue of chasing statistical ghosts. A sudden drop in row count on a Tuesday might be a real data loss, or it might be a holiday in the source system’s country. A spike in nulls might be a schema change, or it might be a new product feature that legitimately changed the data’s shape.
Anomaly detection works best when you have a stable, well-understood baseline and a clear definition of what constitutes an actionable deviation. It works poorly when the underlying data has seasonality, structural breaks, or frequent legitimate changes. If your data has any of those characteristics—and most production data does—you’ll spend more time tuning the anomaly detector than you would have spent writing explicit checks.
If you do use anomaly detection, treat it as a secondary signal, not a primary one. Use it to flag tables or streams that deserve a closer look during a weekly review, not to page someone in the middle of the night. And always pair it with a human-readable explanation of what changed, not just a z-score. “Row count dropped 40% vs. 30-day median” is actionable. “Anomaly score 0.87” is not.
Build a Feedback Loop for Every Alert
The single most effective way to reduce alert fatigue is to make every alert carry a cost for the person who receives it, and a benefit for the person who resolves it. If an alert fires and the on-call engineer’s only option is to acknowledge it and move on, the alert isn’t doing its job. Every alert should have a runbook, a clear owner, and a defined resolution path.
This means you need a feedback loop. When an alert fires, the person who responds should be able to answer three questions:
- Was this alert a true positive or a false positive?
- What action did I take, and did it resolve the underlying issue?
- Should this alert have fired at all, given what I now know?
If the answer to question 3 is “no,” the alert should be tuned, demoted, or deleted. This isn’t a one-time cleanup; it’s a continuous process. Schedule a monthly alert review where the team looks at every alert that fired in the past 30 days and asks whether it earned its place. Alerts that didn’t earn their place get removed. Alerts that fired too late get their thresholds tightened. Alerts that fired too often get their severity downgraded.
This feedback loop is the difference between a monitoring system that improves over time and one that decays. Without it, every new check you add makes the system worse, not better.
Schema Evolution: The Alert Fatigue Multiplier
Schema evolution deserves special attention because it’s the most common source of false-positive data quality alerts in production. When a source system adds a column, renames a field, or changes a data type, your checks will fire—not because the data is wrong, but because the data’s shape has changed. If you don’t have a process for handling schema changes, every schema evolution becomes an alert storm.
The fix isn’t to disable schema checks. The fix is to make schema changes a first-class operational event. When a source system announces a schema change—or when your schema registry detects one—the change should trigger a review process, not a page. The review process should answer: Is this change backward-compatible? Does it break any downstream consumers? Do our existing checks need to be updated to reflect the new schema?
In practice, this means you need a schema change log that is separate from your alerting system. When a schema change is detected, it goes to the schema change log, not to the on-call rotation. A human reviews the change, updates the checks if necessary, and then the checks resume normal operation. This turns schema evolution from a source of alert fatigue into a routine maintenance task.
If you’re using a schema registry like Confluent’s for Kafka streams, you can often automate the detection of schema changes and route them to a review queue. If you’re using a batch pipeline with files landing in S3 or GCS, you can run a lightweight schema inference job on each new batch and compare it to the expected schema. The key is that the comparison result goes to a review queue, not to a pager.
Failure Recovery: Alerts Are Not a Substitute for Resilience
One of the most common mistakes in data quality alerting is using alerts as a substitute for pipeline resilience. If your pipeline fails every time a source system is 10 minutes late, and your response is to add an alert that says “source system is late,” you haven’t solved the problem. You’ve just moved the burden from the pipeline to the on-call engineer.
The better approach is to build resilience into the pipeline itself. If a source system is frequently late, add a retry window or a backfill mechanism. If a stream occasionally emits malformed messages, add a dead-letter queue. If a batch job occasionally runs out of memory, add auto-scaling or checkpointing. Alerts should fire only when the pipeline’s built-in resilience mechanisms have been exhausted and a human decision is required.
This is a hard cultural shift for many teams. It’s easier to add an alert than to fix the underlying pipeline. But every alert you add is a tax on your team’s attention, and attention is the scarcest resource in operational data engineering. Spend the engineering time to make the pipeline resilient first, and reserve alerts for the cases where resilience isn’t enough.
Practical Implementation: A Minimal Alerting Stack
You don’t need a complex observability platform to build a data quality alerting system that doesn’t create fatigue. You need three things: a check runner, a severity classifier, and a routing layer. The check runner can be dbt tests, Great Expectations, a custom SQL script, or a streaming processor like Flink or ksqlDB. The severity classifier is a small piece of logic that maps each check’s output to a severity level based on the business impact you defined earlier. The routing layer sends high-severity alerts to PagerDuty or Opsgenie, medium-severity alerts to a Slack channel, and low-severity alerts to a weekly digest.
Here’s a concrete example. Suppose you run a daily batch pipeline that loads customer orders into a warehouse. You have three checks:
- Freshness check: the
orderstable has data for the current date. Severity: high. If this fails, downstream reports are wrong, and the business needs to know immediately. - Row-count check: the row count for the current date is within 10% of the 30-day median. Severity: medium. If this fails, it might be a data loss or a legitimate business change. A human should investigate during working hours.
- Null-rate check: the
customer_idcolumn is less than 1% null. Severity: low. If this fails, it’s worth a weekly review, but it doesn’t require immediate action.
Each check has a clear owner, a runbook link, and a defined resolution path. The freshness check pages the on-call engineer. The row-count check posts to the #data-quality Slack channel. The null-rate check is logged and included in the weekly data quality digest. This is a minimal system, but it’s a system that respects the team’s attention.
The Maintenance Burden of Checks Themselves
Data quality checks aren’t free. Every check you write is a small piece of software that needs to be maintained. It has a threshold that may need to be updated as the business changes. It has a query that may need to be rewritten as the schema evolves. It has a false-positive rate that may drift over time. If you treat checks as “set and forget,” you’ll end up with a monitoring system that’s as stale as the data it’s supposed to protect.
The maintenance burden of checks is often invisible because it’s distributed across many small tasks: updating a threshold here, fixing a broken query there, suppressing a noisy alert somewhere else. But it adds up. A team that runs 500 checks across 50 tables is spending a significant fraction of its engineering time just keeping the checks alive. That time isn’t spent on improving the pipeline, building new features, or reducing the actual failure rate.
The solution is to treat checks as code, with the same discipline you apply to your pipeline code. Checks should be version-controlled, reviewed, and tested. They should have owners. They should be deleted when they’re no longer useful. And they should be subject to the same cost-benefit analysis as any other piece of infrastructure: does this check prevent enough data quality incidents to justify the time it takes to maintain it?
If you can’t answer that question for a given check, the check shouldn’t exist.
What Good Looks Like: A Case Study in Restraint
Consider a team that runs a streaming pipeline ingesting clickstream data from a mobile app. The pipeline processes about 50 million events per day, and the team initially set up 40 data quality checks: null rates, schema checks, distribution checks, freshness checks, and a handful of custom business rules. Within three months, the team was receiving an average of 15 alerts per day, of which only one or two represented real data quality issues. The on-call engineers started muting the Slack channel. The data quality dashboard became a wall of red that nobody looked at.
The team’s response wasn’t to add more checks or to build a fancier anomaly detection system. It was to cut the number of checks from 40 to 12. They kept the checks that mapped to known failure modes: schema drift, consumer lag, poison messages, and a small number of business-critical null and uniqueness checks. They deleted the rest. They also introduced a severity classification: only schema drift and consumer lag were allowed to page. Everything else went to a daily digest.
The result was a dramatic reduction in alert fatigue. The team went from 15 alerts per day to 2 or 3, and every alert that fired was actionable. The on-call engineers stopped muting the channel. The data quality dashboard became a place where people actually looked for information. The team’s data quality didn’t get worse; it got better, because the signals that mattered were no longer buried in noise.
This is the core lesson: data quality monitoring isn’t about maximizing the number of checks. It’s about maximizing the signal-to-noise ratio of the checks you have. Restraint is a feature, not a bug.
Frequently Asked Questions
How many data quality checks should a production pipeline have?
There’s no universal number, but a useful heuristic is: one check per known failure mode, plus one check per business-critical invariant. If you haven’t observed a failure mode or can’t articulate why a particular invariant matters to a downstream consumer, you probably don’t need a check for it. Most teams find that 10 to 20 well-chosen checks per critical pipeline are more effective than 100 generic checks.
What is the difference between a data quality alert and a pipeline health alert?
A pipeline health alert tells you that the machinery failed: a job crashed, a consumer lagged, a retry exhausted. A data quality alert tells you that the content is wrong: a schema drifted, a null rate spiked, a referential integrity constraint was violated. They should be routed to different channels with different severity levels, because they require different responses. Pipeline health alerts usually need immediate operational action. Data quality alerts often need investigation, not immediate action.
How do I prevent schema evolution from triggering a flood of false-positive alerts?
Route schema change detection to a review queue instead of an alerting channel. When a schema change is detected, a human reviews it, updates the affected checks, and then the checks resume normal operation. This turns schema evolution from an alert storm into a routine maintenance task. If you use a schema registry, you can often automate the detection and routing of schema changes.
Should I use anomaly detection for data quality monitoring?
Anomaly detection can be useful as a secondary signal for flagging tables or streams that deserve a closer look during a weekly review. It’s rarely appropriate as a primary alerting mechanism, because it tends to produce false positives when the underlying data has seasonality, structural breaks, or frequent legitimate changes. If you use it, always pair it with a human-readable explanation of what changed, not just a statistical score.
How often should I review my data quality alerts?
At least monthly. In each review, look at every alert that fired in the past 30 days and ask whether it earned its place. Alerts that didn’t lead to action should be tuned, demoted, or deleted. Alerts that fired too late should have their thresholds tightened. This feedback loop is the single most effective way to prevent alert fatigue from creeping back in.
Next Steps for This Site
This article is part of a broader series on the operational burden of data infrastructure. A natural follow-up is a deep dive on schema evolution strategies for batch pipelines, including how to handle backward-incompatible changes without breaking downstream consumers. Another candidate is a practical guide to building a dead-letter queue for streaming pipelines, with concrete examples from Kafka and Flink. If you have a specific failure mode or alerting pattern you’d like to see covered, the comments are open.