Forum Discussion
Incremental Refresh not working when data is updated
Hi,
I've recently created a test report to see if Incremental Refresh works with PBI Pro connected to MS Blob Storage. The data is imported as .parquet file. There are two reports (exactly the same ones but one with Incremental Refresh and one with just Scheduled refresh).
The problem is that once the new data is loaded to the blob storage, the one with Incremental Refresh did not load the new data but the Scheduled Refresh report did. The steps I took to confiure Incremental Refresh are:
1. Imported the .parquet from the blob storage
2. Created RangeStart and RangeEnd parameter
3. Since there is no date/time column (but DateID column formated like 20210804), I've created a function that converts RangeStart and RangeEnd to match the ID column format
4. Filtered the DateID column and applied RangeStart and RangeEnd as the range
5. Implemented Incremental Refresh Policy for the table
6. Published it and set the Scheduled Refresh
Currently the data shown in the report is for 20210717 - 20210724, and after the refresh it should have changed to 20210717-20210725 but it only shows the data 20210717 - 20210724.
My Questions are:
1. Am I doing something wrong in the steps to configure IR? Do I need Scheduled Refresh set up if the Incremental Refresh is defined before Desktop?
2. I see the warning sign when applying IR. How could I check if query folding does take place? Could this be why it's not working?
Hi v-xulin-mstf ,
I've managed to fix the problem by extending the refresh period from 3 days to 2 months, store period to 3 months! I'll mark this thread as solved. Thanks for the guidance!
5 Replies
Hi aume1,
In any case, your data source must support Query folding if you want to apply Incremental Refresh.
Most data sources that have the concept of a query language support query folding. These data sources can include relational databases, OData feeds (including SharePoint lists), Exchange, and Active Directory. However, data sources like flat files, blobs, and web typically do not.
You can also use the Power Query Diagnostics feature or trace queries by using a tool supported by the data source, like SQL Profiler. If query folding is not occurring, verify the filter logic is included in the query being passed to the data source. If not, it's likely the query includes a transformation that prevents folding.
For more information, please refer:
https://docs.microsoft.com/en-us/power-bi/connect-data/incremental-refresh-overview#query-folding
https://docs.microsoft.com/en-us/power-query/power-query-folding
If you still have some question, please don't hesitate to let me known.
Best Regards,
Link
Is that the answer you're looking for? If this post helps, then please consider Accept it as the solution. Really appreciate!
Hi v-xulin-mstf , Thanks for the reply!
I've changed the data source to Azure SQL DB with the same setting, but to no avail (Even though View Native Query option is active).
I've also been trying to connect DAX Studio to Power BI Service dataset to verify the incremental refresh but keep on encountering an error saying:
with the powerbi://api.powerbi.com/v1.0/myorg/MYWORKSPACENAME
Is there any other methods to check how/why incremental refresh is not working?
Another thing to mention is that when Incremental Refresh is not applied, the data refreshes fine on both Desktop and Power BI service, whereas applying only works on Desktop
Hi aume1,
In your case, maybe the following cause your issue.
You can alao refer this to do troubleshooting.
If you still have some question, please don't hesitate to let me known.
Best Regards,
Link
Is that the answer you're looking for? If this post helps, then please consider Accept it as the solution. Really appreciate!
Hi v-xulin-mstf ,
I've managed to fix the problem by extending the refresh period from 3 days to 2 months, store period to 3 months! I'll mark this thread as solved. Thanks for the guidance!
