Forum Discussion
Large Dataset - Refresh only YTD
Hi Redriver123 ,
Thank you for reaching out to Microsoft Fabric Community Forum regarding the issue you are facing.
Sorry for the delay in response. If the issue is still not resolved, Providing the workaround which might assist you in resolving the issue.
1.Create a dataflow A for historical data and Dataflow B for YTD data. By separating historical and incremental data, avoid refreshing large volumes of unchanged data, saving time and resources.
2.Then,in Power Bi Desktop load load historical and incremental dataflow into the Power Query Editor.
3.Later create parameters for RangeStart and RangeEnd to define the incremental refresh range and close and apply the changes.
4.Publish the report to power bi service and configure the incremental refresh for the dataset.
Please feel free to contact us if you have any further questions. If the response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Hi Redriver123 ,
What is the size of your data in gb?
And how many years of data you are pulling?
Any specific reason you are not using incremental refresh on the entire dataset and provide the incremental range. So if you set it from 1 year, it will refresh only for YTD. And previous years it will not refresh.
If this post helps, please accept this as a solution. Appreciate your kudos.
Thanks,
Pallavi
The Report model size is about 728 MB with 2 import dataflows (historical and current YTD with incremental refresh). Then I append 2 dataflows into a master table in the report. Historical dataflow has 3 years of data about 24 mil records.
Report takes 20 mins to refresh, but YTD dataflow itself takes only 1 min to refresh in the cloud. So I am trying to find a way not to refresh last 3 years (dataflow with history), but seems the process is still re-importing History even when I have disabled "refresh" at dataflow and report level.
Reason I am splitting incremental dataflow into 2 is because I want to avoid re-import of historical last 3 years of data every single refresh.
Hope this was not too confusing.
This is a similar post someone else added which matches the problem. No solution appears to exist.