Forum Discussion

dragospopescu's avatar
dragospopescu
Advocate I
6 months ago
Solved

Can't make MLV refresh incrementally

Hello everyone. Here's my Fabric pickle:   Setup   I have a notebook that writes some tables in the bronze layer using PySpark. The writing of the tables is done using CDF, which is a prerequisit...
data engineering
Notebook
  • ssrithar's avatar
    ssrithar
    5 months ago

    Hi dragospopescu ,

     

    Your MLV is doing too much.

    Between:

    • Overwrite strategy

    • UNION ALL blocks

    • DISTINCT usage

    • Nested LEFT JOINs

    Fabric's incremental engine has no safe way to compute deltas.So it correctly falls back to FullRefresh.

     

    I would do the below changes

     

    1. For bronze layer Instead of replacing 01.01.2026 with 01.02.2026 using mode("append") or MERGE INTO with additional column for ingestion date

     

    2. Instead of distinct create proper dimension tables first and then use only the pre-processed tables in join that would be helpful.ALso remove union to IN

    Split Silver Layer into Multiple MLVs

    Instead of one heavy MLV:

    Do:

    Bronze

    ⬇

    Silver_Base_MLV (no joins, no distinct, no unions)

    ⬇

    Silver_Join_MLV (only joins)

    ⬇

    Gold (aggregations if needed)

    If business logic truly requires: "When February file arrives, January data must disappear completely"

    Then incremental MLV is simply not appropriate. Because that is a batch replacement pattern, not incremental processing.

     

    If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
    If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

ksukhani's avatar
ksukhani
New Member
6 months ago

Hi,

From your description, this is most likely not a configuration or REFRESH command issue, but incremental eligibility falling back to Full due to query pattern or source history.

  • SQL part in the MLV: This is the most likely reason. LEFT JOIN + GROUP BY patterns often make Fabric unable to guarantee incremental correctness, so it silently falls back to Full Refresh. A good quick test is to create a simple single-table MLV (no joins or aggregations) and see if incremental works there.
  • Using PySpark to create tables: The main thing to check is that CDF was enabled before the first data load and that you’re not rewriting tables using overwrite mode (append/merge patterns are safer for incremental scenarios).
  • REFRESH MATERIALIZED LAKE VIEW command: This just executes whatever refresh policy Fabric determines is safe based on lineage, SQL pattern, and change tracking.

If you can share a simplified version of the MLV SQL, it would be easier to pinpoint the exact trigger.