Forum Discussion
Can't make MLV refresh incrementally
Hi dragospopescu ,
Your MLV is doing too much.
Between:
Overwrite strategy
UNION ALL blocks
DISTINCT usage
Nested LEFT JOINs
Fabric's incremental engine has no safe way to compute deltas.So it correctly falls back to FullRefresh.
I would do the below changes
1. For bronze layer Instead of replacing 01.01.2026 with 01.02.2026 using mode("append") or MERGE INTO with additional column for ingestion date
2. Instead of distinct create proper dimension tables first and then use only the pre-processed tables in join that would be helpful.ALso remove union to IN
Split Silver Layer into Multiple MLVs
Instead of one heavy MLV:
Do:
Bronze
⬇
Silver_Base_MLV (no joins, no distinct, no unions)
⬇
Silver_Join_MLV (only joins)
⬇
Gold (aggregations if needed)
If business logic truly requires: "When February file arrives, January data must disappear completely"
Then incremental MLV is simply not appropriate. Because that is a batch replacement pattern, not incremental processing.
If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!
Hi dragospopescu ,
Your MLV is doing too much.
Between:
Overwrite strategy
UNION ALL blocks
DISTINCT usage
Nested LEFT JOINs
Fabric's incremental engine has no safe way to compute deltas.So it correctly falls back to FullRefresh.
I would do the below changes
1. For bronze layer Instead of replacing 01.01.2026 with 01.02.2026 using mode("append") or MERGE INTO with additional column for ingestion date
2. Instead of distinct create proper dimension tables first and then use only the pre-processed tables in join that would be helpful.ALso remove union to IN
Split Silver Layer into Multiple MLVs
Instead of one heavy MLV:
Do:
Bronze
⬇
Silver_Base_MLV (no joins, no distinct, no unions)
⬇
Silver_Join_MLV (only joins)
⬇
Gold (aggregations if needed)
If business logic truly requires: "When February file arrives, January data must disappear completely"
Then incremental MLV is simply not appropriate. Because that is a batch replacement pattern, not incremental processing.
If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!
Thanks for your point of view. I will mark your answer as a solution because most likely it's a combination of over-engineering the MLV and the contradicting requests (incremental refresh, full replace of data).
Even though this will be a closed topic by accepting a solution, I will try my best to come back with the actual technical solution at some point, when I develop something that meets all needs.