The Pipeline Nobody Wanted to Admit Was Broken

There was a time when mentioning ETL in a job posting signaled seriousness. You had data, you had a warehouse, and you had a process for moving the first into the second. That process was ETL: Extract, Transform, Load. It sounded clean. It sounded like engineering. And for a while, it worked well enough that nobody questioned it.

Then somewhere around 2015, the complaints started stacking up. ETL pipelines were brittle. They broke when source schemas changed. They required specialized developers who wrote proprietary transformation scripts nobody else could read. The transformation layer became a bottleneck—data sat in staging tables waiting for someone to fix a truncation error in a column nobody remembered mapping. ETL didn’t scale with data volume, and it didn’t scale with organizational complexity. But the real problem was simpler: ETL assumed you knew what you needed before you looked at the data.

What Went Wrong

The original sin of ETL wasn’t the technology. It was the assumption. Traditional ETL assumed a clean separation between extraction and transformation, where business rules could be defined upfront and applied consistently. In practice, this meant a small team of ETL developers became gatekeepers. Want a new field in the report? File a ticket. Wait three weeks. Discover the field was calculated wrong? File another ticket.

The problems fell into a few recognizable categories:

• Brittle schema dependencies: A source system adds a column, and three pipelines break overnight. Fixing them requires touching scripts that haven’t been updated since someone named Dave wrote them in 2017.
• Opaque transformation logic: Business rules buried in Informatica workflows or SSIS packages that no one can audit without opening a proprietary GUI.
• Slow iteration cycles: Every new data source requires a full ETL design, development, and testing phase before anyone sees a single row.
• Orphaned pipelines: When the person who built the pipeline leaves, the pipeline becomes archaeological evidence rather than maintained infrastructure.

None of this was inevitable. Some organizations ran ETL well for years. But the pattern repeated often enough that when alternatives appeared, the market was ready to declare the entire category obsolete.

Enter ELT

The replacement arrived with a letter rearrangement: ELT. Extract, Load, Transform. The logic was straightforward—move raw data into the warehouse first, then apply transformations using the warehouse’s compute power. Snowflake and BigQuery could handle transformations at scale. Why maintain a separate transformation engine when the warehouse has more compute than you’ll ever need?

dbt made this approach mainstream. Write SQL, version control it, test it, document it. Transformation became software engineering, or at least something adjacent to it. The benefits were real:

• Raw data availability: Analysts can inspect source data before transformation, catching issues early.
• Version-controlled logic: Transformations live in Git, not in a proprietary repository with export limitations.
• Faster iteration: Want a new metric? Write a SELECT statement. Deploy it. Done.
• Transparent calculations: Anyone with SQL access can read the transformation logic.

This was a genuine improvement. But the marketing around ELT sometimes implied that moving the letter “T” solved everything. It didn’t. ELT moved the complexity; it didn’t eliminate it.

The Problems ELT Inherited

Here’s what the ELT proponents don’t always mention: loading raw data first means your warehouse becomes a dumping ground if nobody gets around to the transformation layer. The number of organizations with massive Snowflake bills and a raw schema nobody has touched in months is not small. ELT解决了 the visibility problem but introduced a cost problem. Compute is cheap until it isn’t.

The transformation layer also became fragmented. Instead of one Informatica workflow, you now have forty dbt models maintained by different teams with different testing standards. Centralization shifted to fragmentation. Whether that’s better depends on your governance structure and whether anyone enforces it.

And the original brittleness problem? Still there. Source schema changes still break things. The difference is that now the breakage manifests as a dbt test failure at 2 AM instead of an SSIS error at 6 AM. You’ve changed the timing and the tool, but the fundamental dependency on upstream schema stability remains.

What Actually Replaced ETL

The honest answer is that nothing single replaced ETL. Instead, the concept fragmented into several approaches, each addressing a different aspect of the original problem. If you’re building data infrastructure today, you’re likely combining multiple patterns rather than adopting one replacement.

Change Data Capture (CDC)

CDC tools like Debezium track changes at the source database level and propagate them downstream. Instead of running a nightly batch extract that pulls everything, CDC streams only what changed. This solves the latency problem—data arrives in minutes rather than hours. It also reduces load on source systems, which your database administrators will appreciate.

CDC isn’t without trade-offs. It requires operational discipline at the source. Transaction logs must be available and retained long enough for the capture process to read them. If your source system is a SaaS product you don’t control, CDC may not be an option at all.

Streaming Pipelines

Kafka and similar platforms enabled a different model entirely—continuous data flow rather than batch extraction. Streaming makes sense when freshness matters. Fraud detection, real-time inventory, operational dashboards—these use cases need data within seconds, not hours.

The cost of streaming infrastructure is significant, and most organizations don’t need it for most of their data. A daily batch load handles 80% of analytical workloads perfectly well. The remaining 20% may justify streaming, but only if you can articulate which 20% and why. Building a Kafka cluster because it sounds modern is how you end up with expensive infrastructure nobody monitors properly.

Data Contracts and API-First Approaches

A quieter but potentially more significant shift is the adoption of data contracts—the idea that data producers and consumers agree on schema, freshness, and quality expectations before data moves anywhere. This directly addresses ETL’s brittleness problem. When the source system commits to a contract, downstream pipelines have a stable foundation.

This approach requires organizational maturity and willingness to negotiate. It also works better for internal data sources than external ones. You can’t negotiate a data contract with a vendor API that changes without notice. But for internal systems, contracts enforce the discipline that ETL always assumed but rarely got.

Reverse ETL

Once data lands in the warehouse and gets transformed, someone usually needs it pushed back out to an operational system—CRM, marketing platform, support tool. Reverse ETL tools handle this, closing the loop. It’s a practical recognition that data doesn’t just flow in one direction toward a dashboard. It flows back out to where operational decisions happen.

What Matters More Than the Acronym

The ETL-versus-ELT debate was never the right argument. The real question is simpler: can you trace a number from its source to its destination, explain how it was calculated, and fix it when it breaks?

If the answer is no, your architecture is the problem, not your acronym. I’ve seen ETL systems that were well-documented, tested, and maintained. I’ve seen ELT systems that were an unmaintainable mess. The pattern matters less than the execution.

A few practical principles that survive any architectural trend:

• Know your lineage: If you can’t trace a metric back to its source, you don’t have a pipeline—you have a mystery.
• Test what matters: Not every column needs a uniqueness test. But every financial calculation needs a reasonableness check.
• Document decisions: Why was this transformation written this way? If the answer is “that’s how Dave did it,” you have a documentation problem.
• Monitor actual usage: If nobody queries a table, stop maintaining the pipeline that feeds it. Unused data infrastructure is debt.
• Own your dependencies: Every external system you depend on will change. Plan for it.

The current generation of data tools is better than what came before. But tools don’t substitute for thinking. ETL became a four-letter word not because the pattern was inherently flawed, but because organizations deployed it without discipline and then blamed the acronym when things fell apart. The replacements carry the same risk.

FAQ

Is ETL completely dead?

No. Batch ETL still handles the majority of analytical data workloads in most organizations. The nightly load into the warehouse remains common because most reporting doesn’t require real-time data. What’s changed is that ETL is no longer the only pattern available, and new projects often default to ELT instead. But declaring ETL dead is more about marketing than engineering reality.

Should every organization move to streaming?

Absolutely not. Streaming infrastructure is expensive to build, expensive to maintain, and requires specialized skills most teams don’t have. If your dashboards update once a day and nobody complains, streaming is overkill. Adopt streaming when you have a specific, measurable need for data freshness measured in seconds or minutes—not because a vendor deck says it’s the future.

What’s the biggest risk in replacing ETL with ELT?

Warehouse cost and data governance. Loading raw data first means your storage and compute costs scale with data volume, not with actual usage. And without clear ownership of the transformation layer, the warehouse becomes a swamp of undocumented, untested models. ELT shifts responsibility rather than removing it. If your organization doesn’t have the discipline to maintain transformation logic, moving that logic to SQL doesn’t fix the underlying problem.

It always happens at 3 AM. Not at 2 PM on a Tuesday when you’re sitting at your desk with coffee and a debugger. No, pipelines break when the on-call engineer is asleep, the senior architect is on vacation, and the Slack channel is silent. This is not coincidence. It is the natural consequence of how distributed systems degrade under load, under edge cases, and under neglect.

The Anatomy of a 3 AM Failure

Pipelines don’t break at 3 AM because the universe is cruel. They break at 3 AM because that’s when the accumulated technical debt of the last six months finally tips over into failure. The timeout that was set too aggressively. The disk that’s been filling at 2% per week. The upstream API that silently changed its pagination behavior. None of these things trigger alerts at 10 AM. They wait until the conditions align—until a batch run coincides with a memory leak coincides with a network partition.

I’ve seen teams build elaborate architectures on top of foundations that can’t survive a single process restart. They’ll adopt event-driven microservices with schema registries and service meshes, but nobody bothered to set a retention policy on the message queue. The queue fills up, the consumer falls behind, and suddenly your real-time pipeline is hours behind at 3 AM on a Saturday.

The Usual Suspects

Here’s what typically goes wrong:

Resource exhaustion. Disks fill up. Memory gets consumed. Connection pools max out. These are boring, predictable failures that still catch teams off guard because nobody set up monitoring on something as unglamorous as disk usage trends.

Upstream changes. A vendor modifies their API response format. A source system upgrades and starts producing slightly different JSON. Your pipeline expects one schema and receives another, and the error handling consists of a single try-catch that logs the exception and moves on—silently dropping data.

Dependency failures. Your pipeline depends on a database, a message queue, and an object store. Each has its own failure modes. When the database starts rejecting connections because its connection limit is reached, your pipeline doesn’t gracefully degrade. It falls over.

What Doesn’t Work

Before talking about what to do, let’s cover what doesn’t work. I’ve watched teams try all of these:

Throwing hardware at the problem. Scaling up the cluster feels productive. It’s also expensive and often masks the actual issue—a query that scans an entire table instead of using an index, or a transform that loads everything into memory instead of streaming.

Adding more layers of abstraction. Wrapping your pipeline in an orchestration framework doesn’t make it more reliable. It makes it harder to debug. When something fails at 3 AM, you don’t want to trace through three layers of framework code to find the actual error.

Assuming someone else is handling it. The cloud provider’s SLA covers their infrastructure. It does not cover your misconfigured security group, your application-level bug, or the data quality issue that’s been festering for weeks. Evolutionary architecture doesn’t mean ignoring operational fundamentals.

What Actually Works

1. Know Your Failure Modes

Document every dependency in your pipeline. For each one, ask: what happens when this fails? What happens when it’s slow? What happens when it returns unexpected data?

If the answer to any of these questions is “the pipeline crashes and requires manual intervention,” you have work to do. Your pipeline should degrade gracefully, not catastrophically. A downstream system being unavailable shouldn’t corrupt your data—it should pause processing and retry.

2. Set Meaningful Alerts

An alert that fires at 3 AM should mean something. Not “CPU is at 80%,” but “the ingestion lag has exceeded the two-hour threshold.” Alert on business impact, not on system metrics. If you’re waking someone up, that person needs to know what’s actually wrong, not just that some number crossed a line.

Set up alerts for:

• Processing lag exceeding defined thresholds
• Data freshness dropping below SLA requirements
• Error rates spiking above baseline
• Retry queues growing beyond expected bounds

3. Build Idempotency Into Everything

If your pipeline can’t safely reprocess data, it can’t recover from failures. Idempotency means you can run the same transform twice and get the same result. It means your writes use upserts instead of blind inserts. It means your pipeline can be restarted without producing duplicate records.

Without idempotency, every failure requires manual intervention to figure out where processing stopped, what data was partially written, and how to clean it up before restarting. At 3 AM, under pressure, that manual process is error-prone and slow.

4. Test Failure Scenarios

Chaos engineering gets treated as a trend, but the core idea is sound: deliberately break things in controlled conditions to verify they fail as expected. Kill a database connection mid-process. Fill up a disk partition. Return malformed data from a mock API. Watch what happens.

If you’ve never tested what happens when your primary database becomes unavailable at 3 AM, you’re gambling that the first time it happens, the pipeline will behave correctly. That’s a bad bet.

5. Write Runbooks, Not Just Code

Every known failure mode should have a runbook. Not a wiki page that was last updated eight months ago—a tested, version-controlled document that the on-call engineer can follow at 3 AM without thinking creatively.

A good runbook includes:

• What the alert means in plain language
• Steps to confirm the diagnosis
• Remediation steps, including rollback procedures
• Escalation criteria and contacts

6. Implement Circuit Breakers

When an upstream system starts returning errors, stop hammering it. Implement circuit breakers that detect repeated failures and pause processing until the upstream recovers. This prevents cascading failures and reduces the blast radius of a single dependency going down.

A circuit breaker isn’t complicated. It’s a pattern: track failures, and after a threshold is reached, stop making calls for a cooldown period. Then test whether the downstream system has recovered before resuming normal traffic. It’s the kind of basic engineering practice that gets skipped when teams are focused on feature velocity.

The Boring Fundamentals Matter Most

None of what I’ve described is exciting. Circuit breakers, runbooks, idempotent writes—these aren’t the topics that get presented at conferences. They’re the operational basics that make the difference between a 3 AM page that takes 20 minutes to resolve and one that turns into a four-hour incident.

The teams that sleep through the night aren’t the ones with the most sophisticated architectures. They’re the ones who handled the fundamentals: they set up proper monitoring, they wrote runbooks, they tested their failure scenarios, and they built pipelines that can recover without manual intervention.

Your pipeline will break at 3 AM. The question isn’t whether it will happen—it’s whether you’ll be prepared when it does.

FAQ

What’s the first thing I should do to improve pipeline reliability?

Map your dependencies and failure modes. Before you can fix anything, you need to know where your pipeline is fragile. Document every external system your pipeline depends on, and for each one, describe what happens when it fails. Most teams find gaps in their understanding within the first hour of doing this exercise.

How do I convince leadership to invest in reliability work?

Quantify the cost of downtime. Track the time engineers spend on incident response. Calculate the business impact of data being delayed or incorrect. When leadership sees that a single 3 AM incident costs more in engineer time than a week of reliability work, the investment case becomes clear. Don’t frame it as technical debt—frame it as operational risk with a measurable cost.

Should I use an orchestration tool or build my own pipeline framework?

Use an existing tool. The operational complexity of running a pipeline is already high enough without maintaining custom scheduling, retry logic, and monitoring infrastructure. Tools like Airflow or Prefect handle these concerns so you can focus on your actual data transformations, not on rebuilding task scheduling. The exception is if you have requirements that genuinely can’t be met by existing tools—and that’s rarer than most teams assume.