Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
Jan_Trummel
Helper IV
Helper IV

Incremental Refresh in Dataflow with Folder-Query

Hello to the forum,


I set up a dataflow in a premium workspace. In it I created a folder query, which imports several Excel files from a folder. To improve query performance I now want to enable incremental refresh.


The YouTuber Hoosier BI describes in his video Icremental Refresh with SharePoint Files that the filtering with the two parameters "RangeStart" and "RangeEnd" must be done before the step in which the folder query uses the function to extract  data from the Excel files. However, his video describes the procedure in Power BI Desktop and not in the data flow.

The step "Filtered Rows1" comes before the "Invoke Custom Function1" step (Screenshot from Video)The step "Filtered Rows1" comes before the "Invoke Custom Function1" step (Screenshot from Video)


When I tried to apply this guide to my folder query in the dataflow, I got the following error:

 

Dataflow cannot be saved
Model is invalid because it contains duplicate query names. Error: This document contains a duplicate member "RangeStart".

 

I then simply omitted the two parameters and then set the incremental update. It works perfectly.


My questions are now:
1. So is it correct to omit the "RangeStart" and "RangeEnd" parameters in the incremental update in the data flow?


2. Does incremental updating actually make any sense at all? Yes, Hoosier BI says that the filtering must be done before the function that extracts the data from the Excel files. But without the two parameters, I can't specify when the filtering takes place. So my concern is that Power Query will import all the data from the folder and then apply the logic from the incremental update (which would make that pointless).


What do you think?

3 REPLIES 3
hel
Frequent Visitor

What I meant is if you managed to reduce the refresh time. After some testing, I successfully configured incremental refresh in dataflows as follows:

  1. Activate incremental refresh (without creating the parameters).
  2. Edit the dataflow, remove the automatically generated filter, and create a new one on the date/time column before expanding the function with the automatically created parameters.

I don't know if this is the best way but it worked.

 

Greetings! 

hel
Frequent Visitor

Hi @Jan_Trummel , when you ommited the parameters and set up the incremental refresh, did you see and improve in refresh time? Because when I do it the refresh time increased. I'm also using dataflow and my datasource is files in sharepoint.

Hello @hel , thank you for your answer!

 

What do you mean with "see and improve in refresh time"? How can I improve it?

 

Tank you and have a nice weekend!

Helpful resources

Announcements
July 2024 Power BI Update

Power BI Monthly Update - July 2024

Check out the July 2024 Power BI update to learn about new features.

July Newsletter

Fabric Community Update - July 2024

Find out what's new and trending in the Fabric Community.