Forum Discussion
Update semantic model or Dataflow efficiently
Hi everyone!
I need for Power BI to detect which rows have been modified (based on a key such as DateTime, ClientID, etc.) and update them accordingly once the dataflows are refreshed daily (via Power Automate or scheduled refresh in Power BI). If a row doesn't exist, it should insert it, if it already exists, it should update it.
Currently, I'm dealing with semantic models /dataflows that have millions of rows, and I want to use incremental refresh (or any other solution) to significantly reduce the daily refresh time (potentially running it twice a day) but I've had issues mentioned here https://community.fabric.microsoft.com/t5/Service/Dataflow-incremental-refresh-duplicates-data/m-p/4907013#M292349
If anyone knows how to handle this effectively, that would be great!
Thanks
Power BI incremental refresh is partition-based, so it can refresh partitions efficiently, but it does not “update individual rows” inside an already-loaded partition.
Use:
RangeStart/RangeEnd on a DateTime column (must fold)
optionally Detect data changes using a separate “LastModifiedDateTime” column so only affected partitions refresh
But remember, updates are handled by refreshing the whole partition, not row-by-row upserts.
The duplicate value problem probaly happens because of Range Start/End parameters wrong usage. Example filtering should be like below:
= Table.SelectRows( Source, each [LastModifiedDateTime] >= RangeStart and [LastModifiedDateTime] < RangeEnd )
4 Replies
Hi julsr
Here is some additional information about incremental refresh: https://www.youtube.com/watch?v=RLE0WlZPjqQ
Power BI incremental refresh is partition-based, so it can refresh partitions efficiently, but it does not “update individual rows” inside an already-loaded partition.
Use:
RangeStart/RangeEnd on a DateTime column (must fold)
optionally Detect data changes using a separate “LastModifiedDateTime” column so only affected partitions refresh
But remember, updates are handled by refreshing the whole partition, not row-by-row upserts.
The duplicate value problem probaly happens because of Range Start/End parameters wrong usage. Example filtering should be like below:
= Table.SelectRows( Source, each [LastModifiedDateTime] >= RangeStart and [LastModifiedDateTime] < RangeEnd )Hi julsr
Thank you for posting your query in the Microsoft Fabric Community Forum, and thanks to djurecic and cengizhanarslan for sharing valuable insights.
Could you please confirm if your query has been resolved by the provided solution?If you have any more questions, please let us know and we’ll be happy to help.
Regards,
Microsoft Fabric Community Support Team.
Hi julsr
We have not received a response from you regarding the query and were following up to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.
Thank You.
