Forum Discussion
Scheduled Refresh Policy
- Anonymous2 years ago
Hi debenaire ,
Based on your description, partitioning the data for incremental refresh using the load date seems to be the most efficient approach. This method focuses on new data added to the system without regard to the transaction date, which is consistent with your requirement to only refresh new data without reprocessing historical data in the newly added database. By partitioning by load date, you can refresh only the most recently loaded data, thereby reducing the amount of data processed in each refresh cycle. And when backfilling historical data in a new database, there is no need for a full refresh because the incremental refresh will focus on the load date.
You mentioned that you want to consider using the Detect Data Changes feature to optimize incremental refreshes. This can be especially useful if the data has a column that indicates when a row has been modified. By specifying this column in the incremental refresh policy, Power BI can only refresh rows that have changed since the last refresh, thus further reducing the amount of refreshes.You can refer to the following documentation
Configure incremental refresh and real-time data for Power BI semantic models - Power BI | Microsoft Learn
Incremental refresh for semantic models and real-time data in Power BI - Power BI | Microsoft Learn
Troubleshoot incremental refresh and real-time data - Power BI | Microsoft LearnBest regards,
Albert HeIf this post helps, then please consider Accept it as the solution to help the other members find it more quickly
Hi debenaire ,
Based on your description, partitioning the data for incremental refresh using the load date seems to be the most efficient approach. This method focuses on new data added to the system without regard to the transaction date, which is consistent with your requirement to only refresh new data without reprocessing historical data in the newly added database. By partitioning by load date, you can refresh only the most recently loaded data, thereby reducing the amount of data processed in each refresh cycle. And when backfilling historical data in a new database, there is no need for a full refresh because the incremental refresh will focus on the load date.
You mentioned that you want to consider using the Detect Data Changes feature to optimize incremental refreshes. This can be especially useful if the data has a column that indicates when a row has been modified. By specifying this column in the incremental refresh policy, Power BI can only refresh rows that have changed since the last refresh, thus further reducing the amount of refreshes.
You can refer to the following documentation
Configure incremental refresh and real-time data for Power BI semantic models - Power BI | Microsoft Learn
Incremental refresh for semantic models and real-time data in Power BI - Power BI | Microsoft Learn
Troubleshoot incremental refresh and real-time data - Power BI | Microsoft Learn
Best regards,
Albert He
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly