Data catalogs capture a moment that’s already gone. By the time you’ve documented a table’s schema, ownership, and lineage, an engineer has probably merged a pull request that adds a column, drops a view, or quietly changes what a field actually means. This isn’t a tooling problem. It’s the unavoidable friction between static documentation and infrastructure that never stops shifting. In places where schema changes roll in daily—driven by app migrations, feature releases, or upstream vendor tweaks—the catalog turns into a historical record, not something you can rely on operationally. The real work isn’t building a slicker catalog. It’s accepting that metadata behaves like a stream, not a library, and designing systems that treat it that way.
This piece digs into why the traditional data catalog model is structurally at odds with modern data engineering, what alternatives are taking shape, and how to move your thinking from documentation toward observability. We’ll skip the usual “data democracy” promises and focus on the unglamorous, high-stakes work of keeping production pipelines running when the ground won’t stop moving.
The Half-Life of a Schema Definition
In a typical mid-sized shop, the average lifespan of a table schema keeps shrinking. Continuous deployment means application databases get migrated several times a week. Streaming sources like Kafka topics evolve on their own schedule. Even batch data from third-party APIs can change without a heads-up. A catalog that depends on manual annotation or periodic crawls is always chasing the present. By the time a data steward reviews and certifies a dataset, the underlying structure may have already drifted.
This isn’t theoretical. Picture a financial services firm ingesting trade data from a clearinghouse. The clearinghouse adds a new regulatory field to its feed. The ingestion pipeline, built to be resilient, lands the new field in a semi-structured column. The catalog, refreshed overnight, still shows the old schema. Downstream models that rely on the catalog for discovery miss the new field completely. The data is there, but the catalog has made it invisible. The cost isn’t just a missed opportunity—it’s the slow decay of trust in the data platform.
The False Promise of the Single Source of Truth
“Single source of truth” is one of the most damaging phrases in data engineering. It suggests a static, centralized authority you can consult for definitive answers. In practice, the truth is scattered across query logs, git commits, pipeline configs, and the actual bytes on disk. A catalog that doesn’t reflect this distributed reality isn’t a source of truth—it’s a source of dangerous confidence.
Operational reliability depends on knowing the current state of data, not the state as it was last documented. When an on-call engineer gets paged at 2 a.m. because a critical dashboard broke, they don’t reach for the data catalog. They query the database directly, inspect the pipeline logs, and check the most recent commits. The catalog is often the last place they look, because they’ve learned it’s unreliable. That’s a damning verdict on the whole approach.
The Gap Between Design and Runtime
The core problem is the gap between design-time metadata and runtime metadata. Design-time metadata is what a human declares: “This column is a non-nullable integer representing a user ID.” Runtime metadata is what the system observes: “This column contains strings 3% of the time because of a bug in the upstream service deployed last Tuesday.” A catalog built on design-time metadata is fiction. A catalog built on runtime metadata is a monitoring system.
We have the tools to build the latter. Schema registries, data profiling frameworks, and query log analyzers can all emit continuous signals about the actual shape and content of data. The challenge isn’t technical; it’s organizational. It means accepting that metadata isn’t a product to ship, but a byproduct of operations to capture and query.
Schema Evolution as a First-Class Concern
Schema evolution isn’t a bug. It’s a feature of any living data system. The question isn’t how to prevent it, but how to manage it without breaking downstream consumers. This is where the catalog model fails most visibly. A static catalog treats schema changes as exceptions to document after the fact. An operational approach treats schema changes as events to propagate in real time.
Think about the contract between a producer and a consumer. In a well-architected system, that contract is explicit and versioned. Protobuf, Avro, and JSON Schema all provide mechanisms for schema evolution with compatibility checks. The catalog’s job should be to surface the current contract and its history, not to serve as a mausoleum for deprecated fields. When a producer adds a new optional field, consumers should discover it immediately, not after the next catalog refresh.
Compatibility Enforcement as a Baseline
One of the most practical steps a team can take is to enforce schema compatibility at the producer level. Confluent Schema Registry, for example, lets you set compatibility modes—backward, forward, full—that prevent breaking changes from being written to a Kafka topic. This doesn’t eliminate the need for discovery, but it ensures schema evolution is additive and non-destructive. The catalog then becomes a view into this managed evolution, rather than a desperate attempt to document chaos.
For data warehouses and lakes, the situation is messier. SQL-based systems rarely have built-in schema enforcement. Tools like dbt can offer some guardrails through model contracts and versioned model definitions, but they only cover the transformed layer. The raw ingestion layer remains a wild west. That’s where lightweight profiling and alerting become essential. If a source table’s schema changes, you need to know before the transformation jobs fail, not after.
From Catalog to Observability
The alternative to a static catalog isn’t a better catalog. It’s a shift toward data observability. Observability, in the context of data infrastructure, means being able to ask and answer questions about the current state of the system based on telemetry data. For schemas, this means having a real-time view of the schema of every table, stream, and file in your environment, along with a history of changes and the ability to alert on unexpected modifications.
This isn’t a new idea. Site reliability engineering has been doing this for services for years. We monitor deployments, track configuration changes, and alert on anomalies. Data infrastructure deserves the same treatment. A schema change is a deployment event. It should be tracked in the same change management system, with the same rollback capabilities, and the same communication channels to downstream consumers.
Building a Schema Event Log
The foundation of schema observability is a schema event log. Every time a schema changes—whether through a migration, a new deployment, or an external source—an event is recorded. This event includes the old schema, the new schema, the diff, the timestamp, and the source of the change. This log serves multiple purposes: it enables point-in-time recovery for consumers, provides an audit trail for compliance, and feeds into monitoring and alerting systems.
Implementing this requires hooks into your deployment pipelines and ingestion processes. For databases, you can use DDL triggers or capture schema changes from migration tools like Flyway or Liquibase. For streaming platforms, the schema registry already emits events on schema changes. For file-based sources, you can run a periodic profiler that compares the current schema to the last known state and emits an event if they differ. The key is to make schema change events a first-class citizen in your event-driven architecture.
The Organizational Rot Beneath the Catalog
Technical solutions are necessary but not enough. The catalog problem is also an organizational problem. Data catalogs often become a proxy for ownership disputes, a dumping ground for undocumented datasets, and a checkbox for governance compliance. Teams are incentivized to register their datasets, but not to maintain the metadata. The result is a catalog full of abandoned artifacts, like a wiki nobody updates.
Fixing this requires a shift in incentives. Instead of measuring catalog coverage, measure the freshness and accuracy of metadata. Instead of assigning data stewards responsible for manual curation, make metadata generation an automatic byproduct of the development process. If a team deploys a schema change, the catalog should update automatically. If a team deprecates a table, the catalog should reflect that immediately. The goal is to make the catalog a mirror of reality, not a painting of it.
Ownership as a Runtime Property
Ownership is another area where static catalogs fail. A catalog typically assigns an owner to each dataset. But ownership changes as teams reorganize, people leave, and responsibilities shift. A static owner field becomes outdated quickly. A better approach is to infer ownership from runtime signals: who queries the data most frequently, who deploys changes to the pipeline, who is on-call for the service that produces it. This inferred ownership can be more accurate than any manually maintained list.
This doesn’t mean eliminating human accountability. It means grounding accountability in observable behavior. If a dataset has no clear owner based on runtime signals, that’s a risk indicator. It means the dataset is orphaned and should be flagged for review. This is a more honest and operationally useful approach than a catalog full of names of people who left the company two years ago.
Practical Steps for the Pragmatic Engineer
If you’re responsible for data infrastructure and you recognize this problem, here are concrete steps you can take without buying a new platform or launching a six-month governance initiative.
1. Audit Your Current Schema Drift
Start by measuring the gap between your catalog and reality. Pick a sample of critical tables and compare the cataloged schema to the actual schema in the database. How many discrepancies do you find? How many columns are missing, have changed type, or have different nullability? This audit will give you a baseline for the staleness of your metadata and a compelling argument for change.
2. Implement Schema Change Detection
You don’t need a full observability platform to start detecting schema changes. A simple script that runs INFORMATION_SCHEMA queries against your production databases and diffs the results against a stored snapshot can catch most changes. Store the snapshots in a versioned file or a dedicated table. Send alerts when a change is detected. This isn’t elegant, but it works and it’s better than discovering a breaking change when your ETL jobs fail at 3 a.m.
3. Version Your Schemas with Your Code
If your data pipelines are defined as code, your schemas should be too. Store Avro, Protobuf, or JSON Schema files in the same repository as your pipeline code. Use CI/CD to validate compatibility and deploy schema changes alongside application changes. This couples the schema lifecycle to the software development lifecycle, which is where it belongs.
4. Treat the Catalog as a Read-Only View
Stop trying to make the catalog the source of truth. Instead, make it a read-only view over the actual sources of truth: your schema registries, your git repositories, your database information schemas, and your pipeline metadata stores. The catalog should be a query interface, not a data entry interface. If someone wants to update a description or add a tag, that update should be stored in a version-controlled file next to the schema definition, not in the catalog itself.
What This Means for Your Data Platform
If you accept that the catalog is a view, not a source, then your data platform architecture changes. You invest less in cataloging tools and more in metadata pipelines. You prioritize schema registries, data profiling frameworks, and query log analysis. You build dashboards that show the current state of your data, not a curated snapshot from last week. You alert on schema changes the same way you alert on service outages.
This also changes the conversation with stakeholders. Instead of promising a beautiful catalog where everyone can find everything, you promise a reliable view of the data that actually exists. You trade the illusion of completeness for the reality of accuracy. This is a harder sell, but it’s an honest one. And in data engineering, honesty about the state of the system is the foundation of trust.
Frequently Asked Questions
Why do data catalogs become outdated so quickly?
Data catalogs become outdated because they rely on periodic snapshots of metadata, while the underlying data systems change continuously. In environments with frequent deployments, schema migrations, and external data sources, the catalog is always behind. Manual curation processes cannot keep pace with automated change. The catalog reflects a past state, not the current operational reality.
What is the difference between a data catalog and data observability?
A data catalog is a static inventory of data assets, typically curated by humans. Data observability is a continuous, automated approach to understanding the state of data systems through telemetry, including schema changes, data quality metrics, and lineage. Observability treats metadata as a stream of events, while a catalog treats it as a snapshot. The two can coexist, but the catalog should be a view over observability data, not a separate, manually maintained system.
How can I detect schema changes without buying a new tool?
You can start by writing scripts that query your database’s information schema and compare the results to a stored baseline. For streaming platforms, use the schema registry’s API to track changes. For file-based sources, run a periodic profiler that computes the schema and diffs it against the last known state. Store the results in a simple table and set up alerts for any detected changes. This approach requires no new infrastructure and can be implemented in a few days.
Is it realistic to eliminate manual metadata curation entirely?
Not entirely, but you can drastically reduce it. Business context, such as the meaning of a column or the purpose of a dataset, still requires human input. However, that input should be stored alongside the schema definition in version control, not in a separate catalog UI. Technical metadata—schema, types, nullability, ownership signals—can and should be generated automatically. The goal is to reserve human effort for the metadata that only humans can provide, and to make that effort as lightweight as possible.