Forum Discussion
Incremental Refresh does not load newly loaded data
Hi,
I've recently created a test report to see if Incremental Refresh works with PBI Pro connected to MS Blob Storage. The data is imported as .parquet file. There are two reports (exactly the same ones but one with Incremental Refresh and one with just Scheduled refresh).
The problem is that once the new data is loaded to the blob storage, the one with Incremental Refresh did not load the new data but the Scheduled Refresh report did. The steps I took to confiure Incremental Refresh are:
1. Imported the .parquet from the blob storage
2. Created RangeStart and RangeEnd parameter
3. Since there is no date/time column (but DateID column formated like 20210804), I've created a function that converts RangeStart and RangeEnd to match the ID column format
4. Filtered the DateID column and applied RangeStart and RangeEnd as the range
5. Implemented Incremental Refresh Policy for the table
6. Published it and set the Scheduled Refresh
Currently the data shown in the report is for 20210717 - 20210724, and after the refresh it should have changed to 20210717-20210725 but it only shows the data 20210717 - 20210724.
My Questions are:
1. Am I doing something wrong in the steps to configure IR? Do I need Scheduled Refresh set up if the Incremental Refresh is defined before Desktop?
2. I see the warning sign when applying IR. How could I check if query folding does take place? Could this be why it's not working?
I've solved the issue by extending both storage period and refresh period to 3 and 2 months respectively. I'll mark this thread as solved and thanks for the guidance!
8 Replies
1, yes, you need to do that. Keep in mind that Incremental refresh only works on the SECOND refresh after a meta data update.
2. It's just a note, not a warning. Query folding is not required for incremental refresh to work.
The ONLY true way to check that incremental refresh is working is to connect to the dataset in the Power BI service (for example via DAX Studio or SSMS) and to validate the refresh timestamps on the partitions. If you only see one partition then you know it's not working at all.
Thank you for replying!
Could I ask you what the required steps are to verify the incremental refresh using DAX Studio? I've been trying to connect DAX Studio to PBI Service dataset through powerbi://api.powerbi.com/v1.0/myorg/Word1%20Word2%20Word3%20Word4 (instead of Word1%...Word4, I used my workspace name), but keep getting an error:
Is it because I'm not on Premium Capacity?
No, the error is because you didn't specify the dataset name. (dataset=initial catalog). However, DAX Studio should be able to connect to the workspace first and then give you the list of datasets.
Can you check if you have build permissions on the dataset?
Yes - I have the admin and others who has the access to the workspace has member permissions
Duplicate workspace names maybe?
Dataset connectivity and management with the XMLA endpoint in Power BI - Power BI | Microsoft Docs
No there are no other workspace with the same name.
Trying to connect via:
now gives me an error saying that the URI is not recognised:
Also tried connecting the dataset to OnPremise Data Gateway but Incremental Refresh it didn't work either.
can't see the entire URL. Make sure the prefix is powerbi://
Also for fun you can install the updated AS_OLEDB driver to see if it makes a difference.
I've solved the issue by extending both storage period and refresh period to 3 and 2 months respectively. I'll mark this thread as solved and thanks for the guidance!
