Update semantic model or Dataflow efficiently

[julsr]

Hi everyone!

I need for Power BI to detect which rows have been modified (based on a key such as DateTime, ClientID, etc.) and update them accordingly once the dataflows are refreshed daily (via Power Automate or scheduled refresh in Power BI). If a row doesn't exist, it should insert it, if it already exists, it should update it.

Currently, I'm dealing with semantic models /dataflows that have millions of rows, and I want to use incremental refresh (or any other solution) to significantly reduce the daily refresh time (potentially running it twice a day) but I've had issues mentioned here https://community.fabric.microsoft.com/t5/Service/Dataflow-incremental-refresh-duplicates-data/m-p/4...

If anyone knows how to handle this effectively, that would be great!

Thanks

[v-karpurapud]

Hi @julsr 

We have not received a response from you regarding the query and were following up to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.

Thank You.

[v-karpurapud]

Hi @julsr 

Thank you for posting your query in the Microsoft Fabric Community Forum, and thanks to @djurecic  and @cengizhanarslan for sharing valuable insights.

Could you please confirm if your query has been resolved by the provided solution?If you have any more questions, please let us know and we’ll be happy to help.

Regards,

Microsoft Fabric Community Support Team.

[cengizhanarslan · Accepted solution]

Power BI incremental refresh is partition-based, so it can refresh partitions efficiently, but it does not “update individual rows” inside an already-loaded partition.

Use:

• RangeStart/RangeEnd on a DateTime column (must fold) 

• optionally Detect data changes using a separate “LastModifiedDateTime” column so only affected partitions refresh

But remember, updates are handled by refreshing the whole partition, not row-by-row upserts.

The duplicate value problem probaly happens because of Range Start/End parameters wrong usage. Example filtering should be like below:

= Table.SelectRows(
    Source,
    each [LastModifiedDateTime] >= RangeStart
     and [LastModifiedDateTime] <  RangeEnd
)

[djurecic]

Hi @julsr 

 Here is some additional information about incremental refresh: https://www.youtube.com/watch?v=RLE0WlZPjqQ