Schema evolution is what happens when you change the shape of your data—add a column, rename a field, switch a data type, or retire an attribute—while the system is still running. In data infrastructure work, it sits right at the intersection of storage formats, serialization protocols, and the unspoken contracts between producers and consumers. The trouble isn’t that schemas change. It’s that most teams treat those changes as an afterthought, then act surprised when downstream dashboards break, pipelines stall, and analysts start doubting every number they see. This piece is for the engineers who keep the plumbing working: the ones who know a missing column in a Parquet file can quietly corrupt a week of reports, and a renamed field in an Avro schema can turn a streaming job into a log-spamming mess.
I’ve spent years inside data infrastructure teams where the operational truth is that schemas drift like continental plates. The business adds a new tracking event. The product team renames user_id to customer_id. The analytics group decides revenue should be a decimal, not a float. Each change is small. The cumulative effect on downstream consumers—ETL jobs, materialized views, ML feature stores, real-time alerting—is a slow-motion breakage. The fix isn’t some tool that magically resolves conflicts. It’s a set of practices, formats, and operational habits that treat change as a first-class concern.
Why Schema Evolution Breaks Things
Most data systems are built on a quiet assumption: the schema is a contract. Producers write data in a certain shape, and consumers read it expecting that exact shape. When the contract shifts, consumers fail. How they fail depends on the serialization format and the query engine.
With row-oriented formats like CSV or JSON and no schema registry, a missing column can cause a job to skip records or throw a null-pointer exception. In columnar formats like Parquet or ORC, a renamed column is effectively a new column; the old one still sits in the file metadata, but a consumer looking for the new name finds nothing. In Avro backed by a schema registry, a writer schema that adds a field with a default can be read by consumers using an older reader schema—but only if the default is set correctly and the registry allows the compatibility level you’ve chosen. In Protobuf, adding a field is generally safe if you follow the numbering rules; removing a field is not, because a consumer might still expect that field number.
The common thread: schema evolution is a distributed systems problem. Producer and consumer are decoupled in time. A consumer might read data written hours or days earlier, using a schema version that no longer matches the current writer. Without explicit compatibility rules, the whole thing degrades into a guessing game.
Compatibility Modes: The Foundation of Safe Change
If you’re using a schema registry—and you should be, if you have any kind of shared data infrastructure—the first line of defense is enforcing compatibility modes. Confluent Schema Registry, for example, supports BACKWARD, FORWARD, FULL, and NONE. Which one you pick depends on your consumption pattern.
Backward Compatibility
Backward compatibility means a consumer using a newer schema can read data written with an older schema. This is the default for a lot of teams because it protects existing consumers when producers evolve. You can add a field with a default value, and old data without that field stays readable. You can’t delete a field, because a consumer expecting it would fail on old data that lacks it.
This mode works well when you control the producers and want to let consumers upgrade at their own pace. The operational catch: you have to provide sensible defaults. A nullable field with no default will cause deserialization errors. A non-nullable field with a default that doesn’t match business logic—say, a revenue field defaulting to 0.0—can silently corrupt analytics. I once watched a team add a tax_rate column with a default of 0.0, then spend two weeks debugging why their financial reports were off by exactly the tax amount for historical data.
Forward Compatibility
Forward compatibility means a consumer using an older schema can read data written with a newer schema. It’s harder to achieve and less commonly enforced, but it matters when consumers can’t be upgraded in lockstep with producers—for example, when data lands in a long-lived Parquet table and gets read by dozens of downstream jobs owned by different teams.
To keep forward compatibility, you must never remove a field that existing consumers expect. You can add optional fields, but you can’t change the type of an existing field. Renaming a field is effectively a removal followed by an addition, so it breaks forward compatibility unless your format supports aliases. Avro supports aliases; Protobuf does not. Parquet supports schema merging, but the behavior depends on the query engine. In practice, forward compatibility demands discipline: treat the schema as append-only, and never alter existing fields.
Full Compatibility
Full compatibility is both backward and forward. It’s the safest mode and the most restrictive. You can add optional fields with defaults, but you can’t remove or rename anything. For core datasets that feed many teams—your canonical orders or events tables—full compatibility is often the only sane choice. The cost is that your schema accumulates cruft. Over time, you end up with columns like legacy_user_id, user_id_v2, and customer_id all pointing to the same concept. That’s a documentation and governance problem, not a data-loss problem. I’ll take cruft over breakage any day.
Operational Patterns for Schema Change
Compatibility modes are the safety net. The operational patterns are how you walk across the wire without falling. These patterns assume you’re using a schema registry, versioned schema files, and some form of CI/CD for data pipelines.
Expand-Contract for Renames and Removals
The expand-contract pattern is borrowed from database refactoring. To rename a field, you first expand: add the new field alongside the old one, and write both for a transition period. Update consumers to read from the new field, falling back to the old one if the new field is null. Once all consumers have migrated, you contract: stop writing the old field and eventually remove it from the schema. The transition period must be longer than the maximum lag of any consumer. If a daily batch job reads data from the last 30 days, you need at least 30 days before you can safely remove the old field.
This pattern requires dual-writes and dual-reads, which is tedious. It also requires monitoring to confirm that all consumers have migrated. A practical approach is to log a warning or increment a metric whenever a consumer falls back to the old field. When that metric hits zero for a full retention window, you can proceed with the contract phase.
Schema Versioning in Table Formats
Table formats like Apache Iceberg, Delta Lake, and Apache Hudi have built-in schema evolution features that handle metadata operations atomically. Iceberg, for example, supports add, drop, rename, update, and reorder operations on columns. These operations are committed to the table metadata, and the table history tracks every change. This is a real improvement over directory-based Parquet tables, where a renamed column means rewriting every file.
But table-format schema evolution doesn’t let you stop thinking about consumers. An Iceberg table can rename a column without rewriting data, but a downstream Spark job that references the old column name will still fail. The table format solves the storage-layer problem; it doesn’t solve the contract problem. You still need to communicate changes, version your schemas, and give consumers time to adapt.
Schema Registry as a Source of Truth
A schema registry is more than a compatibility checker. It’s the central nervous system of your data contracts. Every schema change is registered, versioned, and subject to compatibility rules. Consumers can fetch the schema they need by subject and version, or they can rely on the registry to deserialize data using the writer schema and project it into the reader schema.
In practice, this means you should never bypass the registry. Don’t let producers write Avro or Protobuf data without registering the schema first. Don’t let consumers hardcode schemas. The registry is the single source of truth, and treating it that way prevents the drift that happens when teams copy-paste schema definitions into their own codebases.
Downstream Contracts: Views, Not Tables
One of the most effective patterns for insulating consumers from schema changes is to expose data through views rather than raw tables. A view is a stored query that presents a stable interface, even as the underlying table schema evolves. This is a well-established practice in relational databases, but it’s underused in data lake and warehouse environments.
When you create a view, you define the columns, types, and transformations that consumers rely on. If the source table adds a column, the view doesn’t expose it unless you explicitly update the view definition. If a column is renamed in the source table, the view can alias the old name to the new column, preserving backward compatibility. If a column is dropped, the view can provide a default value or derive the column from other fields.
This pattern shifts the burden of compatibility from consumers to the data platform team. That’s where it belongs. The platform team understands the schema lifecycle and can manage views as part of the release process. Consumers get a stable interface, and the platform team gets the freedom to evolve the underlying storage without coordinating with every downstream team.
Versioned Interfaces
For critical datasets, consider maintaining explicit interface versions. Instead of a single view, provide orders_v1, orders_v2, and so on. Deprecate old versions on a published timeline. This is more overhead but gives consumers clear migration paths. It also forces the platform team to think about backward compatibility as a first-class concern, not an afterthought.
Testing Schema Changes
Schema evolution without testing is just hoping nothing breaks. A minimal testing strategy includes:
- Compatibility checks in CI. Every pull request that modifies a schema should run a compatibility check against the previous version. Tools like the Confluent Schema Registry Maven plugin or
avro-toolscan do this. If the change is incompatible, the build fails. - Consumer contract tests. For each downstream consumer, maintain a test that deserializes sample data written with the new schema. This catches issues like missing defaults or type mismatches that compatibility checks might miss.
- Canary deployments. For streaming pipelines, deploy the new schema to a small percentage of traffic and monitor consumer lag and error rates before rolling out fully.
These tests are not optional. They’re the difference between a schema change that goes smoothly and one that wakes you up at 3 a.m.
Handling Breaking Changes
Sometimes a breaking change is unavoidable. A field has to be removed because it contains PII that should never have been stored. A type has to change from string to a structured object. When this happens, the expand-contract pattern is your best tool, but it may not be enough. You may need to coordinate a synchronized upgrade, where producers and consumers switch to the new schema at the same time. This is operationally painful and should be rare.
Another option is to maintain multiple schema versions in parallel, routing consumers to the appropriate version based on their declared compatibility. This is essentially what a schema registry does with reader/writer schema projection, but it requires that all consumers use a client library that supports this feature. In the Kafka ecosystem, this is well-supported. In batch processing with files, it’s harder.
Organizational Habits That Prevent Breakage
Tools and patterns are necessary but not enough. The real work is organizational. Schema evolution is a socio-technical problem. The following habits have helped teams I’ve worked with reduce downstream breakage:
- Schema change announcements. Every schema change, no matter how small, should be communicated to consumers before it goes live. A simple Slack message with the old schema, new schema, and a diff is enough. This gives consumers time to prepare, even if the change is backward-compatible.
- Schema ownership. Every schema should have a clear owner who is responsible for its evolution and for notifying consumers. Ownership can’t be a shared responsibility; that’s the same as no responsibility.
- Deprecation policies. Define how long a deprecated field will be supported before removal. Publish this policy and stick to it. Consumers need to trust that a deprecated field won’t disappear overnight.
- Data contracts. A data contract is an explicit agreement between a data producer and its consumers. It specifies the schema, semantics, SLAs, and deprecation policy. Tools like DataHub and Apache Atlas can help manage contracts, but the contract itself is a social construct. It requires teams to talk to each other.
FAQ
What is the difference between schema evolution and schema migration?
Schema evolution is the process of changing a schema over time while maintaining compatibility with existing data and consumers. Schema migration typically refers to a one-time transformation of data to match a new schema—for example, rewriting a table to add a column to every row. Evolution is ongoing; migration is a point-in-time operation. In practice, evolution often requires migration when a change can’t be made compatible, but the goal is to minimize migrations.
How do I handle schema evolution in a data lake without a schema registry?
Without a schema registry, you’re managing compatibility manually. You can store schema files in version control and enforce compatibility checks in CI. For Parquet files, you can use a table format like Iceberg or Delta Lake to manage schema metadata. The key is to have a single source of truth for schemas and to enforce compatibility rules before data is written. Without these, schema drift is inevitable.
What is the safest way to remove a field from an Avro schema?
The safest way is to use the expand-contract pattern. First, mark the field as deprecated in the schema and set a default value. Update all consumers to stop reading the field. Once no consumer references it, you can remove the field from the writer schema. If you’re using a schema registry with full compatibility, you can’t remove the field directly; you must first change the compatibility mode or use a new subject. Removing a field is a breaking change, so it requires careful coordination.
Can I use Protobuf for schema evolution in streaming systems?
Yes, Protobuf supports backward and forward compatibility if you follow the rules: never change the number of an existing field, only add new fields with new numbers, and avoid removing fields that consumers might still reference. Protobuf doesn’t have built-in schema registry integration like Avro, but Confluent Schema Registry supports Protobuf. The main limitation is that Protobuf doesn’t support aliases, so renaming a field is a breaking change.
Next Steps for Your Data Infrastructure
Schema evolution isn’t a problem you solve once. It’s a practice you build into your daily operations. Start by auditing your current state: do you have a schema registry? Are compatibility modes enforced? Do you have a deprecation policy? If the answer to any of these is no, that’s your starting point. From there, pick one dataset that causes the most downstream breakage and apply the patterns in this article. Measure the reduction in incidents. Use that success to justify the investment in better tooling and processes.
This article is part of a series on operational reliability in data infrastructure. Future pieces will cover monitoring data quality at scale, designing self-healing pipelines, and the role of data contracts in platform engineering. If you have a specific schema evolution war story or a pattern that’s worked for your team, I’d like to hear about it.