Forum Discussion
Can't make MLV refresh incrementally
- 5 months ago
Hi dragospopescu ,
Your MLV is doing too much.
Between:
Overwrite strategy
UNION ALL blocks
DISTINCT usage
Nested LEFT JOINs
Fabric's incremental engine has no safe way to compute deltas.So it correctly falls back to FullRefresh.
I would do the below changes
1. For bronze layer Instead of replacing 01.01.2026 with 01.02.2026 using mode("append") or MERGE INTO with additional column for ingestion date
2. Instead of distinct create proper dimension tables first and then use only the pre-processed tables in join that would be helpful.ALso remove union to IN
Split Silver Layer into Multiple MLVs
Instead of one heavy MLV:
Do:
Bronze
⬇
Silver_Base_MLV (no joins, no distinct, no unions)
⬇
Silver_Join_MLV (only joins)
⬇
Gold (aggregations if needed)
If business logic truly requires: "When February file arrives, January data must disappear completely"
Then incremental MLV is simply not appropriate. Because that is a batch replacement pattern, not incremental processing.
If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!
Hi dragospopescu,
Thank you for reaching out to the Microsoft Fabric Community Forum. Also, thanks to ssrithar, ksukhani, for those inputs on this thread.
Has your issue been resolved? If the response provided by the community member ssrithar, ksukhani, addressed your query, could you please confirm? It helps us ensure that the solutions provided are effective and beneficial for everyone.
Hope this helps clarify things and let me know what you find after giving these steps a try happy to help you investigate this further.
Thank you for using the Microsoft Community Forum.