Data lineage is the recorded path of data from its origin through every transformation, join, filter, and write until it lands in a downstream table, report, or model. Adjacent concepts include data provenance, impact analysis, column-level mapping, and pipeline observability. For mid-career data engineers running batch and streaming pipelines in production, lineage is not a governance slide. It is the difference between a 20-minute root-cause session and a two-day archaeology dig when a schema change breaks a weekly rollup. The hard part is not defining lineage. The hard part is keeping it accurate when schemas evolve, jobs fail and get re-run, and the person who wrote the original pipeline has left the team.
What People Usually Mean by Data Lineage
Most lineage conversations start with a simple question: where did this number come from? In practice, that question splits into at least four different questions. Table-level lineage tells you that fct_orders reads from stg_orders and dim_customers. Column-level lineage tells you that fct_orders.net_revenue is computed from stg_orders.gross_revenue minus stg_orders.discount_amount. Job-level lineage tells you which Airflow DAG or dbt model produced the table and when. Operational lineage tells you which run of that job wrote the specific rows you are looking at, including retries, backfills, and partial failures.
Most tools solve the first two reasonably well. The last two are where production reality lives. A table-level graph that says fct_orders depends on stg_orders is true but nearly useless when a backfill from three weeks ago overwrote a partition with stale exchange rates. The lineage graph did not change. The data did.
Why the First 80 Percent Is Deceptively Simple
If your pipelines are built in dbt, SQLMesh, or a well-structured Airflow repo, you can generate a static lineage graph in an afternoon. Parse the SQL, extract source and target tables, draw the edges. For a single repository with disciplined naming conventions, this works. The graph looks impressive in a demo. Stakeholders nod. The engineering team feels a brief sense of control.
The problem is that static lineage describes intent, not execution. It says what the code should do. It does not say what the code did at 3:14 a.m. on a Tuesday when a retry loop wrote the same batch twice. It does not know that a data engineer manually ran a hotfix script from a laptop because the scheduled job was blocked. It does not know that a streaming job fell behind and consumed events out of order. Static lineage is a map of the roads. Operational lineage is a record of where the trucks actually drove.
Schema Evolution Breaks Lineage Silently
Schema evolution is the most common way lineage graphs rot. A column is renamed in an upstream table. A nested field is promoted to a top-level column. A decimal type is widened to avoid overflow. The downstream pipeline keeps working because the transformation code is resilient or because the change is backward-compatible. The lineage graph, however, now points to a column name that no longer exists.
This is not a theoretical edge case. In a 2023 survey of data professionals, schema changes were among the most frequently cited causes of pipeline failures and data quality incidents. The operational cost is not the schema change itself. It is the silent invalidation of every downstream assumption, including the lineage metadata that was supposed to make those assumptions visible.
If your lineage tool relies on column names as stable identifiers, you are building on sand. Column names are not stable. They are convenient labels that change when business terminology changes, when a new data producer takes over, or when someone finally fixes a naming mistake that has annoyed the team for two years. A lineage system that cannot survive a column rename is a documentation system, not an operational tool.
Failure Recovery Creates Lineage Gaps
Batch pipelines fail. That is normal. What matters is what happens after the failure. A well-run team has retry policies, dead-letter queues, and backfill procedures. Each of those recovery mechanisms creates a new path through the data that the original lineage graph does not capture.
Consider a daily aggregation job that fails at 2 a.m. because a source table was late. The retry runs at 4 a.m. and succeeds. The lineage graph shows one edge from source to aggregate. The operational reality is two attempts, one partial write that was rolled back, and one successful write. If a downstream analyst asks why the aggregate numbers look different from the source for that day, the lineage graph offers no help. The answer lives in the job logs, the retry configuration, and the rollback behavior of the warehouse.
Streaming pipelines make this worse. A streaming job that restarts from a checkpoint may reprocess a window of events. Exactly-once semantics are a property of the processing framework, not of the lineage metadata. If your lineage tool says that events_enriched is derived from events_raw, that is true. It does not tell you that events from 14:02 to 14:07 were processed twice because of a checkpoint restore. The data is correct, or at least consistent with the framework’s guarantees. The lineage is incomplete.
The Maintenance Burden Nobody Budgets For
Lineage is not a one-time implementation. It is a continuous maintenance commitment. Every new pipeline, every refactor, every deprecated table, every migration from one warehouse to another requires updating the lineage metadata. If that update is manual, it will be forgotten. If it is automated, the automation itself becomes a system that can fail.
Teams often underestimate this burden. A lineage initiative starts with enthusiasm. The first few dozen pipelines are mapped. The graph looks useful. Then a reorg moves three teams and their pipelines. A legacy ETL tool is retired. A new streaming source is added. The lineage graph falls behind. Within six months, it is a historical artifact, not an operational tool. The cost of keeping it current exceeds the perceived benefit, and the initiative quietly dies.
The honest tradeoff is this: lineage metadata has value only if it is maintained with the same discipline as the pipelines it describes. That means versioning lineage definitions alongside pipeline code, treating lineage drift as a reviewable issue, and accepting that some percentage of engineering time will go to metadata upkeep. If you are not willing to pay that cost, you are building a demo, not a system.
What Actually Works in Production
After watching several lineage efforts succeed or fail, a few patterns stand out. None of them are free. All of them reduce the maintenance burden enough to make lineage worth keeping.
1. Derive Lineage from Execution, Not Just Code
The most reliable lineage systems I have seen treat the pipeline runtime as the source of truth. They capture the actual inputs and outputs of each job run, including retries, backfills, and manual interventions. This requires instrumentation at the orchestration layer and at the data warehouse layer. It is more work than parsing SQL. It is also the only way to answer the question that actually matters: what happened to this data, not what was supposed to happen.
2. Treat Column-Level Lineage as a Best-Effort Layer
Column-level lineage is valuable for impact analysis, but it is the most fragile part of the system. Column renames, nested schema changes, and dynamic SQL all break it. A pragmatic approach is to maintain table-level lineage as the authoritative graph and treat column-level lineage as a best-effort enhancement that can be stale without invalidating the whole system. When a column-level edge is missing, the system should say so, not guess.
3. Version Lineage Definitions with Pipeline Code
If your pipeline code lives in git, your lineage definitions should live in git too. That means the lineage metadata for a pipeline is updated in the same pull request that changes the pipeline. Reviewers check both. This is the only way I have seen lineage stay current over multiple quarters. It is also the only way to answer questions like “what did the lineage look like before the March refactor?”
4. Accept That Some Lineage Will Be Wrong
This is the hardest lesson for teams that want lineage to be perfect. It will not be. There will be gaps. A manual hotfix will not be recorded. A legacy pipeline will be too expensive to instrument. A vendor tool will not expose the metadata you need. The goal is not a perfect graph. The goal is a graph that is accurate enough to be useful and honest enough to show its own gaps. A lineage system that claims completeness is more dangerous than one that admits uncertainty.
The Cost of Not Doing It
The alternative to lineage is not ignorance. It is slower, more expensive ignorance. When a schema change breaks a downstream report, the team without lineage will trace the problem by reading code, checking logs, and asking colleagues. That process takes hours or days. The team with accurate lineage will trace it in minutes. The difference compounds across every incident, every migration, every compliance audit, and every new team member who needs to understand the data landscape.
There is also a less visible cost. Without lineage, teams become conservative. They avoid refactoring pipelines because they cannot predict the blast radius. They duplicate data instead of reusing it because they do not trust the existing transformations. They build shadow pipelines that are not documented anywhere. Lineage, when it works, is not just a debugging tool. It is an enabler of safe change.
What to Do Next
If you are starting a lineage effort, start small. Pick one critical pipeline. Instrument it end to end. Capture the actual inputs and outputs of each run, including failures and retries. Build the lineage graph from that execution data. Then ask the team that owns the pipeline whether the graph matches their mental model. If it does not, fix the instrumentation before scaling to more pipelines.
If you already have a lineage tool, audit it. Pick a table that has been through a schema change and a backfill in the last month. Trace its lineage in the tool. Then trace it by hand using logs and code. If the two traces disagree, you know where the work is.
Lineage is easy to talk about because the concept is simple. It is hard to implement because production data systems are not simple. The teams that succeed are the ones that treat lineage as an operational system with its own failure modes, maintenance costs, and tradeoffs. The teams that fail are the ones that treat it as a diagram to be drawn once and admired.
Frequently Asked Questions
What is the difference between data lineage and data provenance?
Data lineage describes the path data takes through transformations and pipelines. Data provenance describes the origin and history of a specific data item, including who created it, when, and under what conditions. Lineage is about the pipeline. Provenance is about the record. In practice, the terms overlap, but provenance questions often require operational metadata that lineage graphs do not capture.
Why does column-level lineage break so often?
Column-level lineage relies on stable column identifiers. In production, column names change, nested schemas evolve, and SQL can generate columns dynamically. Each of these changes invalidates column-level edges without necessarily breaking the pipeline. Table-level lineage is more stable because table names change less often and are easier to track through code and logs.
How much engineering time should a team budget for lineage maintenance?
There is no universal number, but a useful rule of thumb is that lineage maintenance should be treated like test maintenance. It is part of the pipeline change process, not a separate project. If lineage updates are not part of the pull request workflow, they will be forgotten. Teams that treat lineage as a separate quarterly cleanup spend more total time and get less reliable metadata.
Can a data catalog replace a lineage system?
A data catalog stores metadata about tables, columns, owners, and descriptions. Some catalogs include lineage features. A catalog alone rarely captures operational lineage because it does not see job runs, retries, or manual interventions. A catalog is a useful complement to lineage, but it is not a substitute for execution-derived lineage.