Forum Discussion
Safest approach for handling schema changes
- 23 days ago
I would say that it will be a good approach to avoid allowing source schema changes to flow directly into the reporting layer.
A common pattern is:
Source → Bronze/Staging → Silver → Gold → Semantic Model
The Bronze layer can preserve the source structure, while the Silver/Gold layers provide a controlled schema for downstream consumers.
For potentially breaking changes such as column removal, renaming, or datatype changes, it is useful to:
- Validate the incoming schema before processing.
- Detect additions, removals and datatype changes.
- Keep the transformation layer independent of unnecessary source-specific changes.
- Test changes in a development/test workspace before production deployment.
- Review the impact on downstream pipelines, tables and semantic models before making breaking changes.
- Use deployment/versioning practices so that schema changes can be introduced in a controlled manner.
Adding a new nullable column is generally less disruptive than renaming/removing an existing column because downstream dependencies may reference the original column name.
For larger Fabric environments, I would also recommend maintaining a metadata/dependency inventory so that when a source column changes, you can identify which pipelines, tables, semantic models and reports could be affected before deploying the change.
The key principle is to detect and assess the schema change before it reaches the Gold/reporting layer, rather than discovering the problem after a production refresh fails.
Even though technically possible, you do not want Fabric to automatically pick up new schemas from sources. One way to implement is medallion architecture. Bronze could pick up new columns but Silver onwards continues to keep old columns so downstream does not break.
Using information schema you could easlily setup a process in place that compares columns between Bronze and Silver and sends you a notification / reports telling you about what has changed and you can react to it on a more organized way.
We have implemented this across all our Lakehouses because we pull data from lot of onprem systems that change schema without notice. Let me know if you need more inputs.