Update semantic model or Dataflow efficiently

[julsr]

Hi everyone!

I need for Power BI to detect which rows have been modified (based on a key such as DateTime, ClientID, etc.) and update them accordingly once the dataflows are refreshed daily (via Power Automate or scheduled refresh in Power BI). If a row doesn't exist, it should insert it, if it already exists, it should update it.

Currently, I'm dealing with semantic models /dataflows that have millions of rows, and I want to use incremental refresh (or any other solution) to significantly reduce the daily refresh time (potentially running it twice a day) but I've had issues mentioned here https://community.fabric.microsoft.com/t5/Service/Dataflow-incremental-refresh-duplicates-data/m-p/4...

If anyone knows how to handle this effectively, that would be great!

Thanks

[cengizhanarslan]

Power BI incremental refresh is partition-based, so it can refresh partitions efficiently, but it does not “update individual rows” inside an already-loaded partition.

Use:

• RangeStart/RangeEnd on a DateTime column (must fold) 

• optionally Detect data changes using a separate “LastModifiedDateTime” column so only affected partitions refresh

But remember, updates are handled by refreshing the whole partition, not row-by-row upserts.

The duplicate value problem probaly happens because of Range Start/End parameters wrong usage. Example filtering should be like below:

= Table.SelectRows(
    Source,
    each [LastModifiedDateTime] >= RangeStart
     and [LastModifiedDateTime] <  RangeEnd
)

[djurecicK2]

Hi @julsr 

 Here is some additional information about incremental refresh: https://www.youtube.com/watch?v=RLE0WlZPjqQ