Forum Discussion
Incremental Refresh doubts
We have Incremental Refresh configured on DATE_TIME_V using TXN_DATE and RangeStart/RangeEnd. DATE_TIME_V is related to LOS_REPORT_V and IVR_SUBHOUR_V through DATE_TIME_KEY. During SQL tracing we see partition-filtered queries against DATE_TIME_V but we do not see date predicates against LOS_REPORT_V/IVR_SUBHOUR_V. Does Power BI push incremental refresh partition filters across relationships during dataset refresh, or must RangeStart/RangeEnd be applied directly on the fact table being refreshed?
If Incremental Refresh is configured only on a Date Dimension table (DATE_TIME_V), but not on the related fact table (LOS_REPORT_V / IVR_SUBHOUR_V), will Power BI still scan the entire fact table during refresh?
We attempted Incremental Refresh directly on IVR_SUBHOUR_V.UPDATE_DATETIME and LOS_REPORT_V.UPDATE_DATETIME, but refreshes were unreliable. We therefore moved RangeStart/RangeEnd filtering to DATE_TIME_V.TXN_DATE. Is there any Microsoft guidance on whether Incremental Refresh should be implemented on the fact table date column instead of a related date dimension?
Does Incremental Refresh partition pruning propagate through relationships from DATE_TIME_V to LOS_REPORT_V/IVR_SUBHOUR_V during dataset refresh, or must the RangeStart/RangeEnd filter be applied directly on the fact table?
Model:
DATE_TIME_V
|
| DATE_TIME_KEY
|
LOS_REPORT_V
DATE_TIME_V.TXN_DATE = RangeStart/RangeEnd
DATE_TIME_V.TXN_DATE = Detect Data Changes
Refresh window = 4 days
Archive = 3 years
Hello manoj_0911
IR filters do NOT propagate through relationships. Your fact tables are doing full scans every refresh.
Incremental Refresh works at Power Query level only. Relationships exist in the semantic model the data source has no knowledge of them during refresh.Add RangeStart/RangeEn` directly in the Power Query of each fact table using TXN_DATE
Table.SelectRows(LOS_REPORT_V, each
[TXN_DATE] >= RangeStart and [TXN_DATE] < RangeEnd)
Do the same for IVR_SUBHOUR_V. Then configure Incremental Refresh on both fact tables using TXN_DATE
If my response helped you, please consider clicking
Accept as Solution ✅ and giving it a Like 👍 – it helps others in the community too.
Thanks,
Connect with me on:
LinkedIn |
Data With Pankaj - YouTubeIn your model, you set up Incremental Refresh on the Date table using TXN_DATE, and it correctly filtered that table during refresh. You could see SQL queries being partitioned on the date dimension, which looked like it was working as expected.
But when you traced the fact tables like LOS_REPORT_V and IVR_SUBHOUR_V, they were still being scanned without any date filters. That’s because Power BI does not push Incremental Refresh filters through relationships during refresh. Relationships only help during report query time, not during data loading.
So even though the Date table was filtered, the fact tables were still processed in full, which caused high CU usage, slow refreshes, and query conflicts.
When you tried moving Incremental Refresh directly to the fact tables using UPDATE_DATETIME, the behavior became unstable, likely due to folding issues or complex transformations breaking the partition logic. That’s why it felt unreliable.
The key realization is that Incremental Refresh must live on the same table that physically holds the filter column. It does not travel across relationships. If the fact table is large, it must be partitioned directly using its own date column, with proper query folding and indexing.
In the end, the Date dimension should stay simple, while each large fact table carries its own Incremental Refresh logic. This shift is what turns a slow, expensive refresh process into a predictable and scalable one.
6 Replies
- pankajnamekar25Super User
Hello manoj_0911
IR filters do NOT propagate through relationships. Your fact tables are doing full scans every refresh.
Incremental Refresh works at Power Query level only. Relationships exist in the semantic model the data source has no knowledge of them during refresh.Add RangeStart/RangeEn` directly in the Power Query of each fact table using TXN_DATE
Table.SelectRows(LOS_REPORT_V, each
[TXN_DATE] >= RangeStart and [TXN_DATE] < RangeEnd)
Do the same for IVR_SUBHOUR_V. Then configure Incremental Refresh on both fact tables using TXN_DATE
If my response helped you, please consider clicking
Accept as Solution ✅ and giving it a Like 👍 – it helps others in the community too.
Thanks,
Connect with me on:
LinkedIn |
Data With Pankaj - YouTube- manoj_0911Kudo Commander
Will there be any difference between scheduled incremental refresh and on demand incremental refresh (which i triggered manually)
- PiotrVerdeHelper I
After Initial refresh (which will create partitions and perform full load) each refresh will follow Incremental Refresh Policy you define for the table (archiving, etc).
It will be the same if you manually click "refresh" or perform scheduled refresh.
- mh2587Super User
In your model, you set up Incremental Refresh on the Date table using TXN_DATE, and it correctly filtered that table during refresh. You could see SQL queries being partitioned on the date dimension, which looked like it was working as expected.
But when you traced the fact tables like LOS_REPORT_V and IVR_SUBHOUR_V, they were still being scanned without any date filters. That’s because Power BI does not push Incremental Refresh filters through relationships during refresh. Relationships only help during report query time, not during data loading.
So even though the Date table was filtered, the fact tables were still processed in full, which caused high CU usage, slow refreshes, and query conflicts.
When you tried moving Incremental Refresh directly to the fact tables using UPDATE_DATETIME, the behavior became unstable, likely due to folding issues or complex transformations breaking the partition logic. That’s why it felt unreliable.
The key realization is that Incremental Refresh must live on the same table that physically holds the filter column. It does not travel across relationships. If the fact table is large, it must be partitioned directly using its own date column, with proper query folding and indexing.
In the end, the Date dimension should stay simple, while each large fact table carries its own Incremental Refresh logic. This shift is what turns a slow, expensive refresh process into a predictable and scalable one.
- VijayPCommunity Champion
manoj_0911 Scheduled refresh , is Just to refresh entire dataset with intervals .
Incremental refresh is to limit the data to refresh to service . So both are not same.
What exactly you want to achieve , Automating the refresh machanism without any time filter or
wanted to refresh only set of transactions in to service. - v-aatheequeCommunity Support
Hi manoj_0911
We wanted to follow up to check if you’ve had an opportunity to review the previous responses. If you require further assistance, please don’t hesitate to let us know.