Forum Discussion
What is wrong with my incremental refresh?
Thb, this incremental load is not going to work here in a proper (designed) way. In the past is was not possible to have one with this data source. This might have changed now?
So if the amount of data from the Sharepoint is too big maybe this might be not the best way to use Sharepoint as a "Database". You are still reading files...this is something completely different like reading from a db. Using Sharepoint like this is often a pain.
If you can try another (more robust) approach.
Another idea...
Maybe you can split your data in different dataflows - each with 6 months of data or so. Maybe you don't need data older than 6 months?
Merge the results afterwards.
Hope you got the idea.
Regards
Hi hansei
Based on the screenshot that the only change made is in the Store rows from the past setting (from 12 months to 18 months), while everything else including the DateTime column and conversions remains the same.
However, when you modify the incremental refresh window in Power BI, the underlying partitions that store data aren’t automatically rebuilt. As a result, the service may continue using cached partitions from the earlier refresh policy (in your case, the 12-month window).
This can lead to unexpected results, such as missing or inconsistent rows between the two runs, even though the data itself hasn’t changed.
To ensure the new 18-month policy takes full effect:
- In Power BI Desktop, turn Incremental refresh Off, apply changes, and then turn it On again with the updated 18-month window.
- Re-publish the dataset to the service.
- In the Power BI service, run a Full refresh once, this forces Power BI to recreate all partitions according to the new refresh policy.
After that, incremental refresh will continue normally and include the full 18-month range as expected.
Reference : Using incremental refresh with dataflows - Power Query | Microsoft Learn
Hope this helps !!
Thank You.
I am not actually using desktop to set incremental refresh. As this forum name and my description indicate, I am setting it in the Power BI service on a dataflow.
Regardless, I cannot "run a Full refresh once". Let me explain further what I trying to accomplish, not just the problem I am running into:
- there is too much data for a full refresh to take place. It will time out in the Power BI service before reading all files.
- I plan to use incremental refresh ongoing, so that it only needs to read changes.
- However, right now, I need to use incremental refresh to read the data in chunks. e.g. Read first 6 months, then next 6 months, then next 6 months. (Or the reverse: read last 6 months, then prior 6 months, then prior 6 months.)
However, the approach I am using had led to the anomalous results explained.
Thb, this incremental load is not going to work here in a proper (designed) way. In the past is was not possible to have one with this data source. This might have changed now?
So if the amount of data from the Sharepoint is too big maybe this might be not the best way to use Sharepoint as a "Database". You are still reading files...this is something completely different like reading from a db. Using Sharepoint like this is often a pain.
If you can try another (more robust) approach.
Another idea...
Maybe you can split your data in different dataflows - each with 6 months of data or so. Maybe you don't need data older than 6 months?
Merge the results afterwards.
Hope you got the idea.
Regards