Forum Discussion
Why does changing the refresh period on an incremental dataflow always result in a full refresh
So we have a large dataflow that set to only refresh the last period but every now and then we have a prior period adjustment that requires prior periods to be adjusted. Now even if its only one extra month it will refresh the entire dataflow which takes 60 times as long (no exaggeration) which is bad enough but then when moving it back to 1 month it will again do a full refresh (so all in all 120x). This is hours of time and I have to stop refreshes of all datasets using the dataflow when this is happening or they will get filled with half missing data or nothing. Its quite the business interuption and I can understand perhaps an increase when increasing the incremental period but not decreasing it as there's no need for a refresh. Surely this is a bug?
ANY meta data change will trigger this unless you use ALM toolkit or other very specific FMLA scripts.
As to why - that was a design decision back in the OLAP cube days. It's all about partition management.
If this is important to you please consider voting for an existing idea or raising a new one at https://ideas.fabric.microsoft.com
6 Replies
ANY meta data change with subsequent publishing of the PBIX to the workspace WILL result in the destruction and recreation of ALL partitions.
If you don't want that, use ALM Toolkit to selectively propagate just the meta data for a given partition.
Thank you, lbendlin , for your response.
Hi AndrewZ,
We appreciate your inquiry through the Microsoft Fabric Community Forum.
Kindly follow the steps mentioned below, which may help in resolving the issue:
- Consider breaking down large dataflows into smaller, linked dataflows. This approach ensures that adjustments in one dataflow do not necessarily trigger full refreshes in others.
- Review the data model for optimisation opportunities. Reducing complexity can significantly help in minimising refresh times.
- Plan dataflow refreshes during off-peak hours to minimise business disruption. Utilise incremental refresh strategies effectively to ensure that only the necessary data is processed during regular operations.
Additionally, please find below some reference links for further guidance:
Using incremental refresh with dataflows - Power Query | Microsoft Learn
Configure incremental refresh for Power BI semantic models - Power BI | Microsoft LearnIf you find our response helpful, kindly mark it as the accepted solution and provide kudos. This will assist other community members facing similar queries.
Thank you.
Hi AndrewZ,
We have not received a response from you regarding the query and were following up to check if you have found a resolution. If you have identified a solution, we kindly request you to share it with the community, as it may be helpful to others facing a similar issue.
If you find the response helpful, please mark it as the accepted solution and provide kudos, as this will help other members with similar queries.
Thank you.
Well I asked why and nobody has given a valid answer you've all just confirmed how it works not why. There is no need for the dataflow to do a full refresh when reducing the number of periods. Likewise a change in a non related metadata should also not trigger a full refresh this is a shortcut approach that will always work without having to check anything. There should at least be a full refresh command on the gui to mimise the amount of times this has to be triggered (minium two for a full refresh)
As for your suggestions
Spliting up the dataflow would work and has been considered but it would make it more difficult for others using the data who may be less tech savy (we have 12 datasets sourcing the one dataflow). This is a finance dataset and not everyone has a stong IT background. Also another nice little bug which makes a dataset table do refresh of a table even if you've set it not to if you happend to append that table to one that is refreshed which makes it pointless.
2 Its a flat file that only has the data necessary so no complexity there
3 All done. This is the important dataflow/dataset with its own dedicated fabric (previously premium) capacity.
As stated above we have some non IT people that need to do this and the ALM toolkit is not an option.
Actually since changing any of the metadata triggers a refresh is there something I can change with minimal impact that will trigger a refresh without having to change the number of periods? Alternatively would chaging the months from 2 to 3 trigger a full refresh and I can leave it at 3 with only a minor increase in refresh time and then change it back to 2 when I next need a refresh or does it do something stupid like trigger a full refresh (and waste all that time) but then only update the last 3 months?
