Forum Discussion
Need help
Hi all, below is my requirement.
Client want to see 5 years of data in the report. They have these data in 3 different tables. (Archieve data - from table 1 in data base A, latest data till before 2 month in table 2 from database B and latest data for last 2 months in table 3 from database B). Now i need to append all this data into one table and then create the visuals from this. The issue is the data volumn is huge (around 40 million records and every day 100000 records will be added). I need to have a incremental refresh . Is it possible to apply incremental refresh for this append query with Power BI Pro licence and shared capasity workspace. If not is there any alternate solution i can use.
Thanks in Advance.
Hi! The answer is - it depends 🙂
In order to utilize incremental refresh you need to be able to ensure query folding. Does the source of the data support query folding. Then, you need to make sure your applied steps in Power Query do not break the query folding. Doing an append may or may not break the folding (if you append as new it will break, if you append two of teh tables to one that you already have, it might not break the folding, but other transformations still need to be considered). Here are some links that will help with the query folding:
Understanding query evaluation and query folding in Power Query - Power Query | Microsoft Learn
Query folding guidance in Power BI Desktop - Power BI | Microsoft Learn
Query folding examples in Power Query - Power Query | Microsoft Learn
1 Reply
Hi! The answer is - it depends 🙂
In order to utilize incremental refresh you need to be able to ensure query folding. Does the source of the data support query folding. Then, you need to make sure your applied steps in Power Query do not break the query folding. Doing an append may or may not break the folding (if you append as new it will break, if you append two of teh tables to one that you already have, it might not break the folding, but other transformations still need to be considered). Here are some links that will help with the query folding:
Understanding query evaluation and query folding in Power Query - Power Query | Microsoft Learn
Query folding guidance in Power BI Desktop - Power BI | Microsoft Learn
Query folding examples in Power Query - Power Query | Microsoft Learn