You spent six months picking a data catalog. Another three rolling it out. The crawlers are finally running. The lineage graphs look spotless in the demo. Then a senior engineer refactors the core ingestion pipeline, renames twelve tables, and deprecates three. Your catalog—still not blessed as “production-ready”—is already a museum piece. This isn’t the tool’s fault. It’s what happens when you forget that a data catalog is a snapshot of a living system, and living systems mutate. The half-life of perfectly accurate metadata in a modern data stack is measured in hours, not weeks. If your catalog strategy doesn’t start from that premise, you’re building a monument to a moment that’s already gone.
The Metadata Decay Curve
Every piece of metadata you capture has a decay curve. Some attributes—like the physical location of a Parquet file on S3—are fairly stable. Others—like the business definition of a customer or the owner of a derived table—rot fast. The trouble is, most catalog initiatives treat all metadata as equally durable. They aren’t.
Picture a typical modern data stack: dbt transforms raw data into models, Airflow orchestrates the runs, and a BI tool sits on top. The dbt project has its own schema.yml files with column descriptions and tests. The BI tool has its own semantic layer with calculated fields and certified datasets. The catalog ingests both, plus table schemas from Snowflake, and tries to merge them into a single view. But the dbt descriptions get updated with every pull request. The BI tool’s metrics change when the business redefines “monthly active user.” The Snowflake schemas shift when a migration script runs. The catalog is always catching up, and it never quite does.
The False Promise of a Single Source of Truth
We’ve been sold the idea that a data catalog will be the “single source of truth” for our data assets. It’s a comforting fiction. In practice, a catalog is a downstream consumer of truth that lives elsewhere: in the git repositories where dbt models are defined, in the orchestration DAGs that execute them, in the wiki pages where analysts document edge cases, and in the Slack threads where engineers debate schema changes. The catalog is a mirror, not the source. And mirrors, as any infrastructure engineer knows, introduce latency and distortion.
The latency is obvious: a crawler runs on a schedule, so there’s always a gap between a change in the source system and its reflection in the catalog. The distortion is subtler. A catalog flattens context. It shows you a table named fact_orders with a column order_status, but it can’t show you the four-hour argument that led to the column being typed as VARCHAR instead of an enum, or the downstream dashboard that will break if you add a new status value. That context lives in pull request comments, architecture decision records, and the collective memory of the team—none of which are easily crawlable.
Operational Metadata vs. Design Metadata
To understand why catalogs fall out of date, we need to separate two categories of metadata. Operational metadata is what the system can observe: table sizes, query frequencies, column data types, lineage derived from SQL parsing. This metadata can be kept reasonably fresh through automated crawling. Design metadata is what humans intend: business definitions, ownership, sensitivity classifications, usage guidelines. This metadata rots because it requires manual upkeep, and manual upkeep does not scale.
The tragedy is that design metadata is what makes a catalog useful. Anyone can run SHOW CREATE TABLE. The value of a catalog lies in answering questions like “Which dataset should I use for weekly active users?” or “Who do I ask if this column looks wrong?” Those answers depend on design metadata, and design metadata is only as good as the last time someone updated it. In most organizations, that update cadence is “when someone complains.”
Ownership as a Distributed System Problem
Catalog tools try to solve the freshness problem with ownership. Assign each asset an owner, send them a notification when the asset’s metadata is stale, and hope they fix it. This is a distributed system problem disguised as a workflow problem. You’re asking dozens or hundreds of people to perform a low-priority maintenance task with no immediate feedback loop. The incentives are misaligned: the person who updates the catalog is rarely the person who benefits from it being accurate. The beneficiary is the analyst three months later who avoids using the wrong table. The owner gets an interruption to their day and no visible reward.
Some teams try to shift ownership upstream by embedding metadata in the code that creates the data. dbt’s schema.yml files are a step in this direction: column descriptions live alongside the transformation logic, and they can be reviewed in the same pull request. This is better than a standalone catalog UI, but it still relies on human discipline. A column description written once and never revisited is only slightly better than no description at all. The real challenge is keeping metadata in sync as the data evolves, and code-embedded metadata doesn’t solve that unless you have a culture of rigorous documentation review—which most teams don’t.
The Schema-on-Read Catalog
If a traditional catalog is always chasing the present, what’s the alternative? One approach is to stop treating the catalog as a static inventory and start treating it as a queryable layer over the actual state of the system. Instead of crawling and storing metadata, you federate queries to the source systems at read time. Want to know the schema of fact_orders? The catalog queries the dbt manifest, the data warehouse’s information_schema, and the BI tool’s semantic layer, then merges the results on the fly.
This pattern—sometimes called a “data discovery platform” rather than a catalog—shifts the freshness burden from the catalog to the source systems. It doesn’t eliminate staleness, but it reduces the surface area. The catalog no longer has its own copy of the metadata that can drift. Instead, it’s a thin aggregation layer that reflects the current state of each source. The tradeoff is query performance and complexity: federated queries are slower and harder to debug than a local index. But for many teams, a slightly slower answer that’s correct beats an instant answer that’s wrong.
Embedding Freshness into the Development Lifecycle
Another strategy is to make metadata freshness a side effect of normal development work, not a separate maintenance task. This means integrating catalog updates into CI/CD pipelines. When a developer opens a pull request that changes a dbt model, the pipeline should validate that the corresponding schema.yml has been updated. When a new table is created in production, the deployment script should register it with the catalog’s API. When a column is deprecated, the catalog should be notified automatically, not through a manual ticket.
This approach treats metadata as a build artifact, not a curated document. It works well for operational metadata that can be derived from code and configuration. It works less well for design metadata that requires human judgment—like writing a useful description or classifying data sensitivity. For those, you still need a human in the loop. But you can reduce the loop’s size by prompting the right person at the right time: when they’re making the change, not weeks later when a crawler notices the discrepancy.
The Maintenance Budget Nobody Allocates
Every architectural decision carries a maintenance burden. When you decide to build a data catalog, you’re committing to an ongoing operational cost: someone must keep it accurate. This cost is rarely budgeted. Teams treat the catalog as a project with a finish line, not a service with an SLO. They celebrate the launch, then move on to the next initiative. Six months later, the catalog is a ghost town of outdated descriptions and broken lineage links, and nobody trusts it anymore.
The honest approach is to define a maintenance budget upfront. How many hours per week will the data platform team spend on catalog upkeep? Which stakeholders are accountable for which domains? What’s the acceptable staleness threshold for different metadata types? If you can’t answer these questions, you’re not ready to deploy a catalog. You’re ready to deploy a prototype that will decay, and you should be explicit about that with your stakeholders. “This catalog will be accurate for approximately three months, after which we’ll need to invest X hours per week to maintain it. If we don’t make that investment, here’s what will happen to data quality.” That’s an honest conversation. Most teams skip it.
Practical Steps for a Living Metadata Layer
Given these constraints, what should a data infrastructure team actually do? The answer isn’t to abandon catalogs—they serve a real need—but to design for impermanence. Here are concrete patterns that acknowledge the decay curve.
1. Tier Your Metadata by Volatility
Not all metadata decays at the same rate. Classify your metadata into tiers based on how quickly it becomes stale. Tier 1: highly volatile (e.g., table row counts, query frequency). Automate collection and accept some staleness. Tier 2: moderately volatile (e.g., column descriptions, owners). Embed in code where possible, and set review gates. Tier 3: stable (e.g., data classification, retention policies). Curate manually with a clear process. This tiering lets you focus maintenance effort where it matters most.
2. Prefer Code-Embedded Metadata
Metadata that lives in the same repository as the code that creates the data has a better chance of staying in sync. dbt’s YAML files, SQL comments, and Python docstrings are all valid homes for metadata. The catalog becomes a consumer of these sources, not the primary store. When a developer changes a model, they change the metadata in the same pull request. The catalog reflects the change on the next ingestion. This isn’t perfect—stale descriptions still accumulate—but it’s better than a standalone UI that nobody visits.
3. Measure and Publish Freshness Metrics
If you want stakeholders to trust the catalog, give them a way to verify its accuracy. Publish a dashboard that shows the percentage of assets with stale metadata, broken lineage links, or unassigned owners. Set a target—say, 90% of Tier 1 assets must have metadata updated within 24 hours—and track it publicly. When the metric drops below the target, that’s a signal to the team that maintenance is needed. This turns catalog freshness from an invisible problem into a visible one.
4. Design for Deprecation
Every asset in your catalog should have a lifecycle: proposed, active, deprecated, removed. When a table is no longer used, the catalog should reflect that. When a column is renamed, the old name should be preserved as an alias for a transition period. This is schema evolution applied to the catalog itself. It’s better to show a deprecated asset with a clear migration path than to delete it and break downstream references. Deprecation is a feature, not a failure.
The Organizational Dimension
Metadata freshness isn’t purely a technical problem. It’s an organizational one. The reason catalogs go stale is that nobody feels responsible for keeping them fresh. Data producers—the engineers building pipelines—see documentation as overhead. Data consumers—the analysts writing queries—see it as someone else’s job. The catalog team, if one exists, can’t possibly keep up with every change across dozens of source systems.
The solution is to distribute ownership, but distribution only works with the right incentives. One pattern is to tie metadata completeness to data SLA compliance. If a pipeline owner is responsible for uptime and data quality, they should also be responsible for keeping the catalog entry for that pipeline current. Another pattern is to make the catalog the primary interface for data access requests. If analysts must go through the catalog to get access to a table, and the catalog shows them the owner, the owner has a strong incentive to keep the entry accurate—otherwise they’ll be fielding questions in Slack all day.
When Not to Build a Catalog
There’s an uncomfortable question that few teams ask: do we actually need a data catalog? For small organizations with a handful of tables and a single data team, the answer is often no. A well-maintained README.md in the dbt repository, combined with a searchable data dictionary in the BI tool, can be more effective than a full catalog. The overhead of deploying and maintaining a catalog may exceed the value it provides.
Even for larger organizations, the question should be: what problem are we solving? If the problem is “analysts can’t find the right data,” a catalog might help—but so might better naming conventions, a curated set of certified datasets, or simply a culture of writing better documentation. If the problem is “we don’t know what data we have,” a catalog is a reasonable answer, but only if you commit to keeping it current. A catalog that’s 60% accurate is worse than no catalog at all, because it breeds false confidence.
Frequently Asked Questions
How often should a data catalog be refreshed?
It depends on the metadata tier. Operational metadata like table schemas and lineage should be refreshed at least daily, and ideally in near-real-time through event-driven ingestion. Design metadata like descriptions and owners can be refreshed less frequently—weekly or monthly—but you should have a process to trigger updates when the underlying data changes. The key is to define freshness SLAs per metadata type and measure compliance, rather than applying a single refresh cadence to everything.
What is the difference between a data catalog and a data discovery platform?
A traditional data catalog ingests and stores metadata in its own repository, creating a static snapshot that requires periodic refreshing. A data discovery platform federates queries to source systems at read time, reducing the staleness problem but introducing latency and complexity. In practice, many modern tools blend both approaches: they cache frequently accessed metadata for performance but can also query live systems for freshness. The distinction matters because it affects your operational burden: a pure catalog requires more maintenance to stay current, while a federated platform requires more engineering to integrate with diverse sources.
How do you measure the ROI of a data catalog?
ROI is notoriously difficult to measure for data catalogs because the benefits—faster data discovery, reduced redundant work, fewer data quality incidents—are hard to quantify. A more practical approach is to measure adoption and freshness: what percentage of your data assets are documented in the catalog, how often are they accessed, and how current is the metadata? If adoption is low or freshness is poor, the catalog isn’t delivering value regardless of what a business case claimed. Track these metrics and be willing to deprecate the catalog if they don’t improve over time.
Can a data catalog replace a data dictionary?
No, and they serve different purposes. A data dictionary is a reference document that defines business terms and their relationships—it’s primarily a design artifact. A data catalog is an inventory of data assets with technical and operational metadata—it’s primarily a discovery tool. A good data strategy needs both, and they should reference each other. The catalog can link to the dictionary for business definitions, and the dictionary can point to the catalog for technical details. But they have different maintenance profiles: a dictionary changes when business concepts evolve, which is relatively slow; a catalog changes when data infrastructure changes, which can be rapid.
The Honest Path Forward
Data catalogs aren’t bad tools. They’re useful when they’re accurate, and they’re accurate when they’re maintained. The problem is that most organizations underestimate the maintenance burden and overestimate the automation capabilities of catalog software. No tool can automatically write a useful column description or determine the sensitivity of a dataset. Those require human judgment, and human judgment requires time and attention.
If you’re building a data catalog, start with the assumption that it will be out of date before it’s finished. Design your processes around that assumption. Automate what you can, embed metadata where it lives, measure freshness relentlessly, and be honest with stakeholders about the ongoing cost. A catalog isn’t a project. It’s a practice. And like any practice, it requires discipline to sustain.
