Forum Discussion
Power BI incremental refresh problem
Hi jtclabaugh
If I understand what you are doing, when the dataset gets created it creates the partitions in the background. you will not have to manage the partitions.
Unless you set your refreshing of the data to multiple days/months it will then have to re-run that data as per the refresh policy?
Let me try and better explain the problem: I have incremental refresh setup - source is SQL (in this case SQL against a databricks spark table). When I run from the desktop, I confirm that 1 *folded* query is run with a where clause that selects AR document date/time >= RangeStart and < RangeEnd. This works perfectly - very efficient. However, when I run from the service to build the partitions (and for subsequent incremental runs), it "works", but an insane number of non-folded limit 1000 queries are fired against the source (note that I also get the folded queries). These take a long time to run and it looks like more are being sent because I have a large number of partitions. From everything I have seen, this is NOT what it's supposed to do. I am expecting to see only folded queries + the select max timestamp for the data change detection. Any ideas?
seems like it's issuing data preview queries like you would get in power query. only happens when running processing for partitions from the service (i.e., tables that aren't partitioned work fine with only the expected folded queries). Here's a visual:
From what I can tell, it's running a data preview query for every step in the power query script * the number of partitions. So if # of partitions is large, an insane number of these queries are run - and the source is big data/ADLS Gen 2 with spark on top
