Forum Discussion
Power BI incremental refresh problem
seems like it's issuing data preview queries like you would get in power query. only happens when running processing for partitions from the service (i.e., tables that aren't partitioned work fine with only the expected folded queries). Here's a visual:
From what I can tell, it's running a data preview query for every step in the power query script * the number of partitions. So if # of partitions is large, an insane number of these queries are run - and the source is big data/ADLS Gen 2 with spark on top
jtclabaugh Did you find a solution for this. We are running PBI over big data much like yourself. We have large PBI models in premium capacity with incremental refresh configured. We are also using the "Detect Data Changes" feature in hope of speeding up the daily refresh. The top 1000 queries are really affecting our refresh tuning. Did you find anyway around this issue?
I'm also interested in a solution.
Since the timestamps are included in the index of the SQL table the incremental refresh query performing well.
The prior query (data preview) during incremental refresh from PBI service is very slow due to select top 1000 instruction
