Forum Discussion

dragospopescu's avatar
dragospopescu
Advocate I
5 months ago
Solved

Can't make MLV refresh incrementally

Hello everyone. Here's my Fabric pickle:

 

Setup

 

I have a notebook that writes some tables in the bronze layer using PySpark. The writing of the tables is done using CDF, which is a prerequisite for the incremental refresh of an upstream MLV using these tables as source.

 

.option(delta.enableChangeDataFeed", "true")

 

I have a separate notebook which is writes an MLV in the silver layer based on the above bronze tables. The code is Spark SQL wrapped in PySpark so that I can use dynamic parameters for lakehouses, schemas, tables and so on, but the MLV is also created using the CDF property (even though I think this only influences any future MLV or table which would use it as a source).

 

CREATE OR REPLACE MATERIALIZED LAKE VIEW [...] TBLPROPERTIES(delta.enableChangeDataFeed=true)

 

The Spark SQL code of the MLV involves select statements, left joins and group by's and I am mentioning this because there seem to be some limitations for incremental refresh based on the SQL code we use according to current Microsoft documentation. BUT I believe this also would be an influence only in the future case of a gold layer using this particular MLV as source.

 

In the silver lakehouse, Optimal Refresh is enabled.

 

I have a third notebook which adds a new row to one of the bronze tables, while also making sure it rewrites the table using PySpark with the CDF property enabled. The notebook is then set up to perform a refresh of the MLV.

 

REFRESH MATERIALIZED LAKE VIEW [...]

 

Problem

 

The resulting metric table view shows RefreshPolicy as being FullRefresh.

 

Help!

 

Can't really figure out what's wrong - Is it the SQL part in the MLV? Is it the fact that I'm using PySpark to create the tables? Is it the REFRESH MLV command which enforces a full refresh instead of an incremental one?

  • ssrithar's avatar
    ssrithar
    5 months ago

    Hi dragospopescu ,

     

    Your MLV is doing too much.

    Between:

    • Overwrite strategy

    • UNION ALL blocks

    • DISTINCT usage

    • Nested LEFT JOINs

    Fabric's incremental engine has no safe way to compute deltas.So it correctly falls back to FullRefresh.

     

    I would do the below changes

     

    1. For bronze layer Instead of replacing 01.01.2026 with 01.02.2026 using mode("append") or MERGE INTO with additional column for ingestion date

     

    2. Instead of distinct create proper dimension tables first and then use only the pre-processed tables in join that would be helpful.ALso remove union to IN

    Split Silver Layer into Multiple MLVs

    Instead of one heavy MLV:

    Do:

    Bronze

    Silver_Base_MLV (no joins, no distinct, no unions)

    Silver_Join_MLV (only joins)

    Gold (aggregations if needed)

    If business logic truly requires: "When February file arrives, January data must disappear completely"

    Then incremental MLV is simply not appropriate. Because that is a batch replacement pattern, not incremental processing.

     

    If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
    If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

6 Replies

  • Hi,

    From your description, this is most likely not a configuration or REFRESH command issue, but incremental eligibility falling back to Full due to query pattern or source history.

    • SQL part in the MLV: This is the most likely reason. LEFT JOIN + GROUP BY patterns often make Fabric unable to guarantee incremental correctness, so it silently falls back to Full Refresh. A good quick test is to create a simple single-table MLV (no joins or aggregations) and see if incremental works there.
    • Using PySpark to create tables: The main thing to check is that CDF was enabled before the first data load and that you’re not rewriting tables using overwrite mode (append/merge patterns are safer for incremental scenarios).
    • REFRESH MATERIALIZED LAKE VIEW command: This just executes whatever refresh policy Fabric determines is safe based on lineage, SQL pattern, and change tracking.

    If you can share a simplified version of the MLV SQL, it would be easier to pinpoint the exact trigger.

  • Hi dragospopescu ,

     

    The problem for the incremental refresh not being cascaded is due to the below reasons

    • GROUP BY + LEFT JOIN pattern not incrementally maintainable this resets the incremental flag

    • Bronze table is being overwritten instead of appended

    • Unsupported aggregation pattern

    • Also is your final notebook doing mode("overwrite") or mode("append")

    If you can share more details of the MLV SQL can provide you the exact details.

     

     

    If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
    If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

     

     

     

     

     

  • Hello ksukhani & ssrithar 

     

    As you can imagine, I can't provide even a simplified version of the codes, but I did my best below in trying to show you how the structure looks like. I think this should help well enough in the context of this post.

     

    Bronze tables

     

    These are created in similar fashion, relying on PySpark for parametrization, dynamic logic for file ingestion, ETL and clean-up, and finally the writing of the tables which is done as below.

     

    df.write.mode("overwrite").format("delta").option("delta.enableChangeDataFeed", "true").partitionBy("Key").saveAsTable(table_name)

     

    The overwrite is there because the input files have timestamps, so if say you already have data from 01.01.2026, when you upload the file for 01.02.2026 you don't need the previous data anymore. Thus, you need to overwrite the existing data in the lakehouse.

     

    Silver MLV

     

    Below is the structure/skeleton for the MLV, as mentioned at the beginning.

     

    sql_statement = f"""
    
    CREATE OR REPLACE MATERIALIZED LAKE VIEW mlv_nane
    TBLPROPERTIES (delta.enableChangeDataFeed=true)
    
    AS
    
    SELECT 
        columns 
    FROM (
        SELECT columns FROM table WHERE conditions
        UNION ALL
        SELECT columns FROM table WHERE conditions
        UNION ALL
        SELECT columns FROM table WHERE conditions
        UNION ALL
        SELECT columns FROM table WHERE conditions
    ) AS base_table
    
    LEFT JOIN (SELECT DISTINCT columns FROM table) AS join_1 
        ON base_table.id = join_1.id
    
    LEFT JOIN (SELECT DISTINCT columns FROM table) AS join_2 
        ON base_table.id = join_2.id
    
    LEFT JOIN (SELECT DISTINCT columns FROM table) AS join_3 
        ON base_table.id = join_3.id
    
    LEFT JOIN (SELECT DISTINCT columns FROM table) AS join_4 
        ON base_table.id = join_4.id
    
    LEFT JOIN (SELECT DISTINCT columns FROM table) AS join_5 
        ON base_table.id = join_5.id
    
    """
    
    spark.sql(sql_statement)

     

    Testing

     

    In the notebook I created for testing, the logic is something like this:

     

    from pyspark.sql import Row
    
    mock_row = Row( columns and values )
    
    mock_df = spark.createDataFrame([mock_row])
    
    mock_df.write.format("delta").option("delta.enableChangeDataFeed", "true").mode("append").saveAsTable(table_name)

     

    Note: table_name from Testing is the same table_name from the Bronze layer.

    • ssrithar's avatar
      ssrithar
      Super User

      Hi dragospopescu ,

       

      Your MLV is doing too much.

      Between:

      • Overwrite strategy

      • UNION ALL blocks

      • DISTINCT usage

      • Nested LEFT JOINs

      Fabric's incremental engine has no safe way to compute deltas.So it correctly falls back to FullRefresh.

       

      I would do the below changes

       

      1. For bronze layer Instead of replacing 01.01.2026 with 01.02.2026 using mode("append") or MERGE INTO with additional column for ingestion date

       

      2. Instead of distinct create proper dimension tables first and then use only the pre-processed tables in join that would be helpful.ALso remove union to IN

      Split Silver Layer into Multiple MLVs

      Instead of one heavy MLV:

      Do:

      Bronze

      Silver_Base_MLV (no joins, no distinct, no unions)

      Silver_Join_MLV (only joins)

      Gold (aggregations if needed)

      If business logic truly requires: "When February file arrives, January data must disappear completely"

      Then incremental MLV is simply not appropriate. Because that is a batch replacement pattern, not incremental processing.

       

      If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
      If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

      • dragospopescu's avatar
        dragospopescu
        Advocate I

        Thanks for your point of view. I will mark your answer as a solution because most likely it's a combination of over-engineering the MLV and the contradicting requests (incremental refresh, full replace of data).

         

        Even though this will be a closed topic by accepting a solution, I will try my best to come back with the actual technical solution at some point, when I develop something that meets all needs.

  • Hi dragospopescu,

    Thank you for reaching out to the Microsoft Fabric Community Forum. Also, thanks to ssritharksukhani,  for those inputs on this thread.

    Has your issue been resolved? If the response provided by the community member ssritharksukhani,  addressed your query, could you please confirm? It helps us ensure that the solutions provided are effective and beneficial for everyone.

    Hope this helps clarify things and let me know what you find after giving these steps a try happy to help you investigate this further.

    Thank you for using the Microsoft Community Forum.