Dataflow filtering but still full refresh

[joshua1990]

Hi

we have a dataset / report that is linked to a huge dataflow (150 columns and like 5MM rows).

I have like as a second step to filter only for the current week based on a date field but in the desktop application the refresh takes still the same time as a full refresh.

Why is filtering within power query not impacting the refresh from dataflows?

[lbendlin]

The filter query doesn't fold. It is only applied after all the data is downloaded.

[joshua1990]

So it is normal that the refresh from dataflows takes so long?

[lbendlin · Accepted solution]

Dataflows are just glorified CSV files in Azure Blob storage (They do usually ingest fast though - maybe you do too many transforms?).

If you want to streamline the process then cut out the middle man (the dataflow) and implement Incremental Refresh of your Semantic Model table from a foldable source.