Forum Discussion
Large Dataset - Refresh only YTD
Hi Redriver123 ,
Thank you for reaching out to Microsoft Fabric Community Forum regarding the issue you are facing.
Sorry for the delay in response. If the issue is still not resolved, Providing the workaround which might assist you in resolving the issue.
1.Create a dataflow A for historical data and Dataflow B for YTD data. By separating historical and incremental data, avoid refreshing large volumes of unchanged data, saving time and resources.
2.Then,in Power Bi Desktop load load historical and incremental dataflow into the Power Query Editor.
3.Later create parameters for RangeStart and RangeEnd to define the incremental refresh range and close and apply the changes.
4.Publish the report to power bi service and configure the incremental refresh for the dataset.
Please feel free to contact us if you have any further questions. If the response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Hi Redriver123
When working with large datasets in Power BI Dataflows—especially scenarios involving a large historical dataset alongside a smaller, frequently updated incremental dataset—the goal is to optimize refresh performance and avoid unnecessary processing. In your case, you have Dataflow A holding the historical data (about 20 million rows) with refresh disabled, and Dataflow B for Year-To-Date (YTD) incremental data that refreshes quickly. However, when appending these two in a report or combined Dataflow, both dataflows end up refreshing, causing long refresh times.
To efficiently handle this, it’s best to separate the historical and incremental dataflows clearly and avoid appending them inside Power BI or Dataflow at refresh time, which can trigger both to refresh. Instead, maintain Dataflow A as a static, rarely refreshed data source. For Dataflow B, enable incremental refresh properly so only the new or changed data is processed daily. Then, in your report or dataset, combine these two sources by merging or appending the data at the query or model level without forcing a refresh of the historical dataflow. Another common approach is to materialize the combined dataset in a Power BI dataset or a data warehouse/table that aggregates historical and incremental data outside the dataflow refresh process, thereby decoupling the refresh schedules.
If using Power BI Premium or Fabric capacities, consider leveraging large dataset storage or incremental refresh policies at the dataset level rather than in dataflows. This allows the historical data to remain stable while incremental data updates occur efficiently, reducing refresh time drastically. Overall, the key is to design the refresh architecture so that the large static data is refreshed infrequently (e.g., monthly or quarterly), and only the incremental portion refreshes daily, avoiding unnecessary refresh cascades and improving overall performance for all reports relying on that master dataset.
Thank you for such thoughtful response. I am happy to see someone actually read my post.
1) " in your report or dataset, combine these two sources by merging or appending the data at the query or model level without forcing a refresh of the historical dataflow."
That's the core of the problem. When these are combined in power query, this kicks off the full refresh process of importing the historical data from (non refreshing) dataflow.
Unless you suggest using a DAX to create a virtual table for UNION both historical and incremental refresh. Maybe, this could work, but I would imagine performance issues.
2) Another common approach is to materialize the combined dataset in a Power BI dataset or a data warehouse/table that aggregates historical and incremental data outside the dataflow refresh process, thereby decoupling the refresh schedules."
This would defeat the whole purpose of using incremental refresh feature in PBI. Also, setting this up in datawarehouse would force the import process of all 20 million rows. Which we do not want.