Forum Discussion
Incremental refresh for dataflow takes as much time as full refresh
Can you define a view in the you SQL source that defines the query, including the CTEs, that you call in your data flow?
Resolver IFrom your screenshots it looks like the Incremental refresh, at least in the way it is set up now, is not actually loading a portion of the data during incremental load.
As we see in your screenshots, in both full and incremental refreshes the 'Rows Processed' column shows pretty much the same number of rows.
This could be, as Deku mentions, due to query folding not working. Or it could be that the date field to which you have set the incremental load is affecting a lot more rows that you initially expect.
How do I know if the query is folding ? here's what I see in the dataflow editor:
- Deku
Super User
The red green colour strip indicates query folding
https://learn.microsoft.com/en-us/power-query/step-folding-indicators
You can see your second step breaks folding as it has a red indicator
thank you, I believe it's because i'm using a custom select. I searched online and the only thing I found to enable query folding with custom select is to add a step with Value.NativeQuery(...) and put the sql query inside
is there any other way to enable query folding apart from this workaround ? this workaround doesn't seem to allow the use of ctes.
