Forum Discussion
Large Dataset - Refresh only YTD
Hi Redriver123 ,
Thank you for reaching out to Microsoft Fabric Community Forum regarding the issue you are facing.
Sorry for the delay in response. If the issue is still not resolved, Providing the workaround which might assist you in resolving the issue.
1.Create a dataflow A for historical data and Dataflow B for YTD data. By separating historical and incremental data, avoid refreshing large volumes of unchanged data, saving time and resources.
2.Then,in Power Bi Desktop load load historical and incremental dataflow into the Power Query Editor.
3.Later create parameters for RangeStart and RangeEnd to define the incremental refresh range and close and apply the changes.
4.Publish the report to power bi service and configure the incremental refresh for the dataset.
Please feel free to contact us if you have any further questions. If the response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Super UserHi Redriver123
When working with large datasets in Power BI Dataflows—especially scenarios involving a large historical dataset alongside a smaller, frequently updated incremental dataset—the goal is to optimize refresh performance and avoid unnecessary processing. In your case, you have Dataflow A holding the historical data (about 20 million rows) with refresh disabled, and Dataflow B for Year-To-Date (YTD) incremental data that refreshes quickly. However, when appending these two in a report or combined Dataflow, both dataflows end up refreshing, causing long refresh times.
To efficiently handle this, it’s best to separate the historical and incremental dataflows clearly and avoid appending them inside Power BI or Dataflow at refresh time, which can trigger both to refresh. Instead, maintain Dataflow A as a static, rarely refreshed data source. For Dataflow B, enable incremental refresh properly so only the new or changed data is processed daily. Then, in your report or dataset, combine these two sources by merging or appending the data at the query or model level without forcing a refresh of the historical dataflow. Another common approach is to materialize the combined dataset in a Power BI dataset or a data warehouse/table that aggregates historical and incremental data outside the dataflow refresh process, thereby decoupling the refresh schedules.
If using Power BI Premium or Fabric capacities, consider leveraging large dataset storage or incremental refresh policies at the dataset level rather than in dataflows. This allows the historical data to remain stable while incremental data updates occur efficiently, reducing refresh time drastically. Overall, the key is to design the refresh architecture so that the large static data is refreshed infrequently (e.g., monthly or quarterly), and only the incremental portion refreshes daily, avoiding unnecessary refresh cascades and improving overall performance for all reports relying on that master dataset.