Forum Discussion

codeautomation's avatar
codeautomation
Icon for Advocate II rankAdvocate II
2 days ago

How Do You Handle Schema Changes in Microsoft Fabric Lakehouses Without Breaking Reports?

Hi everyone,

I'm interested in how teams are managing schema evolution in Microsoft Fabric Lakehouses, particularly when source systems frequently introduce new columns, change data types, or rename existing fields.

A common challenge in enterprise data engineering is ensuring that changes in upstream systems don't unexpectedly break downstream Delta tables, Spark notebooks, SQL queries, or Power BI semantic models.

Microsoft Fabric supports schema evolution for Delta tables, but allowing schema changes automatically isn't always the safest approach for production workloads.

I'd be interested in hearing how other data engineers handle a few scenarios:

  1. Schema Drift: Do you allow automatic schema evolution using mergeSchema, or do you prefer explicitly approving changes before ingestion?
  2. Data Contracts: Are you using formal data contracts between source systems and Fabric Lakehouses to prevent unexpected breaking changes?
  3. Pipeline Testing: How do you validate schema compatibility before executing production ingestion and transformation jobs?
  4. Downstream Dependencies: What's your approach to identifying affected notebooks, SQL endpoints, and Power BI semantic models before changing a table schema?
  5. Recovery Strategies: When a schema change breaks an existing workflow, do you roll back the change, introduce a versioned table, or maintain backward-compatible views?

I'm particularly interested in practical solutions that work across Bronze, Silver, and Gold layers in a medallion architecture.

One question I'd especially like to explore: Should enterprise data engineering teams prioritize flexible schema evolution or strict data contracts as their Fabric environments scale?

Would love to hear what has worked in real production environments, including any lessons learned from unexpected schema changes.

Thanks!

4 Replies

  • v-achippa's avatar
    v-achippa
    Icon for Community Support rankCommunity Support

    Hi codeautomation​,

    Thank you for reaching out to Microsoft Fabric Community.

    The best approach to use is flexible ingestion in Bronze, with stable schemas in Silver and Gold. Use mergeSchema for approved schema additions in fabric lakehouses. Keep Gold’s schema stable for power bi and test changes in a nonproduction workspace before deploying. For production workloads, I recommend validating schema changes before applying them to avoid unexpected changes breaking downstream pipelines and reports.

    For reference, follow below documentation:
    Schema evolution for Delta tables - Microsoft Fabric | Microsoft Learn

    Thanks and regards,
    Anjan Kumar Chippa

  • Hi codeautomation

    I'd split it by layer instead of choosing one side. Bronze takes drift, silver enforces, gold is the contract.

    In bronze, let new columns in with mergeSchema. It only adds columns, it never drops or retypes anything. In silver, keep Delta's default schema enforcement and add a check before the write that compares the incoming columns and types with what you expect. If they differ, fail the run and let a person approve the change. Just don't turn on spark.databricks.delta.schema.autoMerge.enabled there.

    In gold, treat a rename or drop as two steps: add the new column, move the notebooks and models over, then drop the old one. Both need column mapping, so turn it on early:

    ALTER TABLE sales SET TBLPROPERTIES ('delta.columnMapping.mode' = 'name')

    New columns don't appear in semantic models on their own, so adding them is safe. Renames, drops and type changes are what break reports and the SQL analytics endpoint, so warn the report owners first. This page covers the options: https://learn.microsoft.com/en-us/fabric/data-engineering/delta-lake-schema-evolution

  • ShivekMaharaj's avatar
    ShivekMaharaj
    Icon for Community Champion rankCommunity Champion

    Hi codeautomation​,

    For production Fabric environments, I would not use one schema-evolution policy across Bronze, Silver, and Gold.

    In Bronze, I am comfortable allowing additive drift in many cases. Microsoft's Delta schema evolution guidance recommends mergeSchema for new columns, and that works well when Bronze is meant to preserve the incoming source shape.

    In Silver, I would make the contract much stricter. This is where I would validate expected columns, data types, nullability, and key fields before writing. A new source column may be accepted into Bronze automatically, but I would not let it silently become part of a curated Silver contract without review.

    Gold is where I would be most conservative because that is usually what semantic models, SQL consumers, and reports depend on. Microsoft's guidance specifically warns that renamed or dropped columns can break notebooks, semantic models, reports, and the SQL analytics endpoint.

    For renames and drops, I would prefer a backwards-compatible transition rather than changing the contract in place. For example, add the new column, keep the old one temporarily, migrate downstream consumers, and only remove the old field once usage has been cleared.

    If you expect renames or drops, Microsoft's Delta column mapping guidance is also worth planning for early. Column mapping allows metadata-only rename and drop operations, but it upgrades the Delta protocol, so every downstream reader should be checked for compatibility before enabling it.

    For type changes, I would distinguish safe widening from semantic changes. Fabric now supports type widening for compatible changes such as some integer-to-larger-number conversions, but even those can affect downstream consumers, so I would still test them before production.

    So my general rule would be:

    • Bronze = flexible ingestion and additive drift
    • Silver = explicit schema contract and validation
    • Gold = backwards-compatible changes only, with dependency review

    I would also keep schema checks in CI/CD or in the pipeline before the write, rather than discovering breaking changes after a semantic model refresh fails.

    Fabric's own schema evolution guidance recommends testing changes in development and reviewing downstream dependencies before applying them to production, which is probably the most important part of the whole strategy.

    For me, flexibility is useful at ingestion boundaries. Stability matters more as the data moves closer to consumers.

    AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

  • Hi codeautomation,

    Adding a few Fabric-specific points to the answers above, mostly from the Delta Lake interoperability doc:

    1. Renames without rewriting data. Turn on column mapping for Silver/Gold tables, and a rename becomes a metadata-only change:

    ALTER TABLE silver_customer SET TBLPROPERTIES (
      'delta.columnMapping.mode' = 'name',
      'delta.minReaderVersion' = '2',
      'delta.minWriterVersion' = '5');
    ALTER TABLE silver_customer RENAME COLUMN cust_nm TO customer_name;

    The SQL analytics endpoint and Direct Lake semantic models can both read name-mapped tables. A rename still breaks anything that uses the old name (views, measures, report fields), so I treat it as a breaking change.

    2. Check who writes the table. Python notebooks (delta-rs, Polars, DuckDB) don't support column mapping or type widening with the deltalake versions Fabric currently pins. If a Python notebook writes to that table, keep the writes on Spark or don't enable these features. Enabling column mapping also upgrades the table protocol, so any external reader has to support it too.

    3. mergeSchema in Bronze only. In Silver/Gold, compare the incoming schema before you write and fail on surprises:

    expected = set(spark.table("silver_customer").columns)
    incoming = set(df.columns)
    missing, new_cols = expected - incoming, incoming - expected
    if missing:
        raise ValueError(f"Missing columns: {missing}")
    if new_cols:
        print(f"New columns to review: {new_cols}") # add with ALTER TABLE ... ADD COLUMNS after review

    4. Gold is a contract. Allow only additive changes there. For a breaking change, build a v2 table, move the consumers over, then retire v1.

    Docs: learn.microsoft.com/fabric/fundamentals/delta-lake-interoperability

    Hope this helps!