Forum Discussion
Safest approach for handling schema changes
- 24 days ago
I would say that it will be a good approach to avoid allowing source schema changes to flow directly into the reporting layer.
A common pattern is:
Source → Bronze/Staging → Silver → Gold → Semantic Model
The Bronze layer can preserve the source structure, while the Silver/Gold layers provide a controlled schema for downstream consumers.
For potentially breaking changes such as column removal, renaming, or datatype changes, it is useful to:
- Validate the incoming schema before processing.
- Detect additions, removals and datatype changes.
- Keep the transformation layer independent of unnecessary source-specific changes.
- Test changes in a development/test workspace before production deployment.
- Review the impact on downstream pipelines, tables and semantic models before making breaking changes.
- Use deployment/versioning practices so that schema changes can be introduced in a controlled manner.
Adding a new nullable column is generally less disruptive than renaming/removing an existing column because downstream dependencies may reference the original column name.
For larger Fabric environments, I would also recommend maintaining a metadata/dependency inventory so that when a source column changes, you can identify which pipelines, tables, semantic models and reports could be affected before deploying the change.
The key principle is to detect and assess the schema change before it reaches the Gold/reporting layer, rather than discovering the problem after a production refresh fails.
Hi Selcii-16,
This depends on how your source system is changing. In general there's no safe way to handle removed columns. If a source removes a column that a report depends upon, then the report will break. I suppose you could always keep the column null on your side, but then the report may no longer be accurate as any visual that references that column will display nothing.
Adding columns is easy, you can set up your ETL to overwrite the schema in spark and that will add the new columns to your bronze/silver tables, and then I would recommend you leave them there until they are needed for something. Adding the columns to a semantic model is a manual process to my knowledge.
If columns are renamed, you'd need a way to identify what the old name was so you can remap the column.
I'd argue that any source that frequently renames or drops columns is a bad source.