Incremental load Dataflow

Hi Hussein_charif1 ,

Thank you for reaching out to the Microsoft Community Forum.

 

Hi Pragati11  and lbendlin , Thank you for your prompt responses.

 

Hi Hussein_charif1 , 

Incremental refresh only reprocesses partitions that overlap the refresh window. Older partitions remain static.

“Detect data changes” does not scan your full historical data. It evaluates the refresh window and decides whether to refresh those partitions based on the max value of the change-detection column within that window. It does not trigger reprocessing of older partitions (Jan‑2025) that are outside the refresh window. The user edited records from Jan 2025, their PostingDate is historical, so those rows live in older partitions. With a 1‑week refresh window, those partitions were never revisited, no update appeared.

 

Please try below workaround.

 

1. Use LastModifiedDateTime as the filter column, This captures any row modified recently, regardless of its original posting date. In the dataflow’s Incremental refresh settings:

 

Filter by: LastModifiedDateTime
Refresh window: choose a window that matches how often changes occur..
Bucket size: Day or Hour if you need granularity.
Detect data changes: You can turn it off when using LastModifiedDateTime as the filter column, since the filter itself targets recent modifications, or keep it on to further skip refreshes when nothing changed in the window.

2. Ensure query folding to the source: Incremental refresh relies on pushing the date filter down. If folding breaks, Power BI/Dataflow may pull all rows and filter them client‑side, defeating incremental refresh. For Business Central, the OData stack used by some BC APIs doesn’t fully support folding for date filters,RangeStart/RangeEnd aren’t applied server‑side, causing long refreshes. Consider OData.Feed connector with simple query steps and apply the LastModifiedDateTime filter early. Avoid transformations before the filter that can break folding. Check View Native Query on the filtering step; if greyed out, folding is broken.

 

Please follow below steps.

1. Open your Dataflow Gen2 --> Edit the query you want to refresh.
2. Keep the query folding, push filters down as early as possible; check View Native Query.
3. Right‑click the query --> Incremental refresh:

Filter column: LastModifiedDateTime
Extract data from the past: pick a window that matches change frequency.
Bucket size: Day.
Only extract new data when max value changes.

4. Publish and run a full refresh once to reconcile historic updates.

 

Please refer below links.

Incremental refresh in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

Using incremental refresh with dataflows - Power Query | Microsoft Learn

Solved: Incremental refresh data changes on dataflow - Microsoft Fabric Community

Solved: Incremental refresh in Dataflows using Business Ce... - Microsoft Fabric Community

 

I hope this information helps. Please do let us know if you have any further queries.

 

Regards,

Dinesh