Incremental Refresh Last Step, can be moved?

[GuilleATT]

Hi,

I have applied the incremental refresh to my dataset with the hope of time refresh reduction but the result is that it takes the same time or more.

My dataset has lots of transformations and merges with other queries and when checking on the transformation steps I see that there is a new one that filters rows between RangeStart and RangeEnd... my two question are then:

--> Could I move this row filtering to the beginning of the dataflow so that all calculations are performed only in the rows that changed? Otherwise, I'm afraid I won't get to the benefits of the incremental refresh. 

--> Do I need different columns for Datetime column to filter by and the "Detect data changes"? why? where is the "Detect data changes reflected in the Query?

Thank you in advance!

Guillermo

[brunoctesser]

I really don't get it. The solution was supposed to improve refresh times by filtering out rows that don't need to get transformed, and yet it is only possible to filter unnecessary rows in the last step, AFTER the whole data is transformed? I thought that somehow the filtering was being applied before the steps, but I noticed the refresh times weren't being much faster.

[lbendlin · Accepted solution]

@GuilleATT Your post is inconsistent. You talk about datasets but then show details for a dataflow.

The filter step can be applied earlier in a Power Query script on Power BI Desktop (ie for a dataset)

You can move the step forward in the dataflow Power Query editor on the service but the moment you save the dataflow it will rearrange the steps again and put the filter step back at the end.  That is really, really annoying and one of the reasons I don't use incremental refresh for dataflows.

I really hope they remove that silly limitation at some point (together with the silly extra error check step that they insist on adding)