Forum Discussion
Can't make MLV refresh incrementally
- 6 months ago
Hi dragospopescu ,
Your MLV is doing too much.
Between:
Overwrite strategy
UNION ALL blocks
DISTINCT usage
Nested LEFT JOINs
Fabric's incremental engine has no safe way to compute deltas.So it correctly falls back to FullRefresh.
I would do the below changes
1. For bronze layer Instead of replacing 01.01.2026 with 01.02.2026 using mode("append") or MERGE INTO with additional column for ingestion date
2. Instead of distinct create proper dimension tables first and then use only the pre-processed tables in join that would be helpful.ALso remove union to IN
Split Silver Layer into Multiple MLVs
Instead of one heavy MLV:
Do:
Bronze
⬇
Silver_Base_MLV (no joins, no distinct, no unions)
⬇
Silver_Join_MLV (only joins)
⬇
Gold (aggregations if needed)
If business logic truly requires: "When February file arrives, January data must disappear completely"
Then incremental MLV is simply not appropriate. Because that is a batch replacement pattern, not incremental processing.
If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!
Hi dragospopescu ,
The problem for the incremental refresh not being cascaded is due to the below reasons
GROUP BY + LEFT JOIN pattern not incrementally maintainable this resets the incremental flag
Bronze table is being overwritten instead of appended
Unsupported aggregation pattern
- Also is your final notebook doing mode("overwrite") or mode("append")
If you can share more details of the MLV SQL can provide you the exact details.
If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!