Six months of mapping every table, column, and transformation in the warehouse. The catalog is finally “done.” You schedule a demo for the analytics team. Midway through, a data engineer mentions that dim_customer was deprecated last Tuesday and replaced by a view that unions three regional sources. The lineage graph on the screen is already wrong. This isn’t an execution failure. It’s a failure of the premise—that a data catalog can ever be a static artifact. A catalog is a snapshot of a living system, and living systems don’t pause for documentation.
For teams managing operational data stores, streaming pipelines, or schema-on-read architectures, the catalog’s decay rate is measured in hours, not sprints. The core tension sits between discovery (finding what exists) and fidelity (accurately describing what exists). Most catalog initiatives optimize for the first and quietly sacrifice the second. This article digs into the structural reasons catalogs fall out of date, the maintenance burden that gets hidden in project plans, and the architectural patterns that accept staleness as a first-class design constraint rather than a bug.
The Half-Life of Metadata
Metadata decays. A column rename, a schema migration, a deprecated source—each change widens the gap between the catalog’s description and the system’s reality. The half-life of a metadata entry depends on the volatility of the underlying asset. In a batch warehouse with quarterly releases, a table definition might hold for weeks. In a Kafka-backed microservice environment where producers evolve schemas independently, the half-life can be under 24 hours.
Consider a typical event-driven architecture. The OrderPlaced topic has 14 producer services, each managed by a different team. The schema registry enforces compatibility, but the semantic meaning of fields drifts. The discount_amount field originally captured promotional discounts. The logistics team later started using it for damage-related adjustments. The catalog still says “promotional discount.” The catalog is technically correct—the schema hasn’t changed—but operationally misleading. This is the difference between schema drift and semantic drift, and most catalogs only detect the first.
Why Automated Crawlers Cannot Save You
The standard response to staleness is automation. Schedule a crawler to scan your data lake, warehouse, and schema registries every night. Update the catalog with new tables, columns, and lineage. This sounds reasonable until you examine the failure modes.
Crawler Blind Spots
Crawlers see what the system exposes. A table that exists but is no longer used still appears as a first-class asset. A critical dataset generated by a dbt model that runs on a cron schedule inside a Kubernetes pod might be invisible because the crawler only scans the warehouse’s information schema. The lineage between a Kafka topic and the materialized view that consumes it is lost unless the crawler can parse both the stream processor’s code and the warehouse’s query logs. Most cannot.
The Ownership Problem
Even when a crawler detects a new column, it cannot assign ownership. The column user_segment_v2 appears in the fact_events table. Who created it? The growth team’s data scientist, who ran a one-off enrichment job that wrote back to the production table. The catalog now shows an unowned column. The data engineering team gets paged when it breaks a downstream model. The growth team does not know the catalog exists. Automation surfaced the asset without surfacing the accountability structure. This is worse than not having the column documented at all, because it creates a false sense of governance.
The Maintenance Budget Nobody Funds
Building a catalog is a project. Maintaining a catalog is an operational commitment. Projects get headcount and deadlines. Operations get pager rotations and a line item in the infrastructure budget. Most organizations treat the catalog as a project. They staff a data governance team for 12 months, buy a tool, run the crawlers, and declare victory. Then the governance team is reassigned, and the catalog begins its slow death.
The maintenance burden is not trivial. It includes:
- Schema change triage: Every new column, deprecated table, or type change must be reviewed. Is the change intentional? Does it break downstream consumers? Who should be notified?
- Ownership reconciliation: When a team reorganizes, their assets need new owners. This is a manual process unless your org chart is machine-readable and kept current—it is not.
- Semantic validation: Does the documented description still match reality? This requires talking to the producer, which does not scale.
- Lineage repair: When a pipeline changes, the catalog’s lineage graph must be updated. If the pipeline is defined in code (dbt, Airflow, custom scripts), the catalog must parse that code. If the pipeline is a manual process, the catalog will never be accurate.
Each of these tasks requires human judgment. The catalog tool can flag anomalies, but it cannot decide whether a new column is a bug or a feature. That decision requires context: the product roadmap, the data model’s intended evolution, the team’s current priorities. No tool has that context. The humans who do are already busy.
Treating the Catalog as a Living System
If the catalog cannot be a static artifact, it must be a living system. This means designing it with the same operational rigor you apply to your production databases. The catalog has an SLA. It has an on-call rotation. It has a recovery procedure when it drifts too far from reality.
Define Acceptable Staleness
Not all staleness is equal. A table description that is three months out of date might be acceptable if the table is a stable, batch-loaded dimension. A streaming topic’s schema that is three hours out of date is a liability. Segment your assets by volatility and define staleness thresholds for each tier. Monitor those thresholds. Alert when they are breached. This is the same approach you use for data freshness in your warehouse; apply it to your metadata.
Embed Catalog Updates into Development Workflows
The only sustainable way to keep a catalog current is to make updates a side effect of the work that causes the changes. When a developer merges a PR that adds a column to a production table, the catalog should update automatically. This requires the catalog to be code-driven, not UI-driven. Tools like dbt already generate manifest.json files that describe models, columns, tests, and lineage. Your catalog should consume that manifest as its source of truth, not a separate manual entry.
For assets that are not code-defined—legacy tables, manually created views, third-party data feeds—accept that the catalog will be stale. Flag these assets explicitly. Label them “unverified” or “last confirmed” with a date. Make the staleness visible so consumers can apply their own risk tolerance.
Ownership as a First-Class Contract
Every asset in the catalog should have a defined owner, and that ownership should carry operational responsibilities. The owner is not just a name in a field. The owner is the person who gets paged when the asset’s freshness SLA is breached. The owner is the person who approves schema changes. If you cannot assign an owner who accepts these responsibilities, the asset should be marked as “unowned” and treated as deprecated by default. This creates a natural incentive for teams to either claim ownership or stop using unowned assets.
Schema Evolution Patterns That Break Catalogs
Certain schema evolution patterns are particularly hostile to catalog accuracy. Recognizing them helps you decide where to invest maintenance effort—or where to accept that the catalog will be a rough approximation.
Wide Tables with Frequent Column Additions
In organizations where analysts are empowered to create columns directly in production tables, the catalog becomes a graveyard of undocumented fields. Each new column is a liability: no description, no owner, no lineage. The catalog’s completeness metric drops daily. The fix is not better crawling; it is a process change that gates column additions through a code review and documentation step. This slows down analysts, which is the tradeoff. Be explicit about whether speed or documentation fidelity matters more for each table.
Schema-on-Read Systems
Data lakes and document stores allow schema to be applied at query time. The same Parquet file can be read with different schemas by different teams. A catalog that captures one schema is capturing a partial truth. The catalog must either support multiple schema projections per asset or clearly state which projection it documents. Most catalogs do neither, leaving consumers to discover the mismatch at query time.
Multi-Tenant Event Streams
When a single Kafka topic carries events from multiple producer teams, the schema is often a union of all possible fields. Any given producer uses a subset. The catalog shows the union, which is technically correct but operationally useless for a consumer who wants to know which fields are actually populated. The catalog needs to track field population rates and surface them alongside schema definitions. This is a monitoring problem, not a documentation problem.
What a Catalog Can Realistically Do
Given these constraints, a data catalog is not a source of truth. It is a discovery aid with a known error rate. Treating it as anything else leads to broken pipelines and eroded trust. A realistic catalog provides:
- Approximate lineage that is directionally correct but may miss edges or include deprecated paths.
- Schema snapshots with timestamps, so consumers can assess freshness themselves.
- Ownership metadata that is as current as the last org chart update, with clear “last verified” dates.
- Usage statistics (query frequency, read rows) to help consumers distinguish live assets from zombie tables.
Anything beyond this—semantic descriptions, business glossaries, data quality scores—requires ongoing human investment. If you cannot fund that investment, do not build those features. A catalog that claims to have accurate business descriptions but actually has stale ones is worse than a catalog that honestly says “description unavailable.”
Operational Patterns for Living Catalogs
If you accept that your catalog will always be partially out of date, you can design processes that minimize the damage. These patterns come from teams that have run catalogs in production for years, not months.
Embedded Deprecation
When a table or column is deprecated, the deprecation should be visible in the catalog immediately, not after the next crawl. This requires a push mechanism: the system that owns the asset must notify the catalog at deprecation time. For code-defined assets, this can be a CI/CD hook. For manually managed assets, it requires discipline—or acceptance that the catalog will be wrong.
Consumer-Driven Corrections
Allow catalog consumers to flag inaccuracies. A data analyst who discovers that a column’s description is wrong should be able to submit a correction with one click. That correction goes to the asset owner for approval. This distributes the maintenance burden across the organization and surfaces issues that automated checks miss. The key is making the feedback loop short: if corrections take weeks to process, nobody will submit them.
Staleness as a Feature
Rather than hiding staleness, expose it. Show the “last verified” date prominently on every asset page. Color-code assets by freshness. Let consumers filter out assets that have not been verified in 90 days. This shifts the burden from the catalog team to the asset owners: if you want your dataset to be discoverable, you must keep it current. If you do not care, the dataset fades from view.
FAQ
Why not just automate everything with crawlers?
Crawlers can detect structural changes—new columns, dropped tables, schema modifications—but they cannot detect semantic drift, assign ownership, or validate that a description still matches reality. Automation reduces the maintenance burden but does not eliminate it. The remaining gap requires human judgment, and that judgment must be funded as an ongoing operational cost, not a one-time project.
How do I convince leadership that catalog maintenance is an ongoing cost?
Frame it in terms they already understand: technical debt. A catalog that is not maintained accumulates metadata debt, just as a codebase accumulates technical debt. The interest payments are broken pipelines, incorrect analyses, and eroded trust in the data platform. Present a specific example from your own organization where stale metadata caused a measurable problem—a failed report, a wrong business decision, an incident. Tie the maintenance cost to the risk of recurrence.
Should we even build a data catalog if it will always be out of date?
Yes, but with realistic expectations. A catalog that is 80% accurate is still valuable for discovery, especially in large organizations where analysts cannot keep track of every dataset. The key is to be honest about the 20% inaccuracy. Do not market the catalog as a source of truth. Market it as a map: useful for navigation, but not guaranteed to show every pothole. And invest in the processes that keep the map current, or accept that it will gradually become a historical artifact.
What is the difference between a data catalog and a schema registry?
A schema registry (like Confluent Schema Registry) enforces schema compatibility at the producer level and is typically tightly coupled to a streaming platform. A data catalog is a broader discovery tool that spans multiple systems. The schema registry is authoritative for the schemas it manages; the catalog is a best-effort aggregation. Confusing the two leads to expectations that the catalog can enforce governance, which it cannot without the operational machinery of a registry.
Next Steps for This Publication
This article is part of a series on the operational realities of data infrastructure. Future pieces will examine the maintenance burden of feature stores, the hidden costs of real-time pipelines, and the organizational patterns that make schema evolution survivable. If you maintain a data platform and have stories of catalog decay—or strategies that worked—I would like to hear from you. Reader questions and war stories shape the editorial direction here.