Incremental Refresh in Dataflow with Folder-Query

[Jan_Trummel]

Hello to the forum,

I set up a dataflow in a premium workspace. In it I created a folder query, which imports several Excel files from a folder. To improve query performance I now want to enable incremental refresh.

The YouTuber Hoosier BI describes in his video Icremental Refresh with SharePoint Files that the filtering with the two parameters "RangeStart" and "RangeEnd" must be done before the step in which the folder query uses the function to extract  data from the Excel files. However, his video describes the procedure in Power BI Desktop and not in the data flow.

The step "Filtered Rows1" comes before the "Invoke Custom Function1" step (Screenshot from Video)

When I tried to apply this guide to my folder query in the dataflow, I got the following error:

Dataflow cannot be saved
Model is invalid because it contains duplicate query names. Error: This document contains a duplicate member "RangeStart".

I then simply omitted the two parameters and then set the incremental update. It works perfectly.

My questions are now:
1. So is it correct to omit the "RangeStart" and "RangeEnd" parameters in the incremental update in the data flow?

2. Does incremental updating actually make any sense at all? Yes, Hoosier BI says that the filtering must be done before the function that extracts the data from the Excel files. But without the two parameters, I can't specify when the filtering takes place. So my concern is that Power Query will import all the data from the folder and then apply the logic from the incremental update (which would make that pointless).

What do you think?

[hel]

What I meant is if you managed to reduce the refresh time. After some testing, I successfully configured incremental refresh in dataflows as follows:

1. Activate incremental refresh (without creating the parameters).
2. Edit the dataflow, remove the automatically generated filter, and create a new one on the date/time column before expanding the function with the automatically created parameters.

I don't know if this is the best way but it worked.

Greetings! 

[hel]

Hi @Jan_Trummel , when you ommited the parameters and set up the incremental refresh, did you see and improve in refresh time? Because when I do it the refresh time increased. I'm also using dataflow and my datasource is files in sharepoint.

[Jan_Trummel]

Hello @hel , thank you for your answer!

What do you mean with "see and improve in refresh time"? How can I improve it?

Tank you and have a nice weekend!