Forum Discussion
Post ProdigyDoes incremental refresh cascade down to referenced/merged queries?
We're currently loading 6 or so years worth of data from Excel files in Sharepoint into the AuditWF table. The output from that query is then referenced and merged with 2 separate tables, resulting in the AUFI_1 and AUFI_2 tables. I would like to apply incremental refresh rules to the AuditWF (and potentially the 2 other tables it is merging with in the AUFI_1 and AUFI_2 queries).
The question is, would it actually apply the rules not only in the query where I applied it but also to any queries referencing the query where the incremental refresh rules were applied?
Incremental refresh does not cascade. You need to configure it separately on AuditWF, AUFI_1, and AUFI_2.
One thing to flag: SharePoint Excel files don't support query folding, so incremental refresh won't work as expected. It will pull the full data every time. Move to a SQL or Lakehouse source if you want incremental refresh to actually work.Hi olimilo,
Thank you for posting your query in the Microsoft Fabric Community Forum.Incremental refresh policies are configured at the individual table/query level and do not automatically propagate to referenced or merged queries. Therefore, you do not need to duplicate AuditWF, but if AUFI_1 and AUFI_2 are also loaded as separate tables and require incremental refresh behavior, you'll need to configure an incremental refresh policy for each of those tables separately.
Regarding your question about using SharePoint file metadata such as Modified Date, filtering based on metadata can help reduce the amount of data processed, depending on how the query is evaluated. However, incremental refresh relies on query folding to push filters to the data source. If the source does not support query folding, filtering might not be applied at the source level, which can reduce the effectiveness of incremental refresh.
In general, incremental refresh delivers the best performance when query folding is available, allowing filters to be applied at the source.
For more information, please refer to the official documentation:
Using incremental refresh with dataflows - Power Query | Microsoft Learn
Incremental refresh in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
Thanks, Kedar_Pande for sharing valuable insights.
Best regards,
Ganesh Singamshetty
5 Replies
- Kedar_Pande
Super User
Incremental refresh does not cascade. You need to configure it separately on AuditWF, AUFI_1, and AUFI_2.
One thing to flag: SharePoint Excel files don't support query folding, so incremental refresh won't work as expected. It will pull the full data every time. Move to a SQL or Lakehouse source if you want incremental refresh to actually work.
olimilo> You need to configure it separately on AuditWF, AUFI_1, and AUFI_2.
So in essence, I would have to duplicate the AuditWF and apply the inc refresh rules for each instance?
> SharePoint Excel files don't support query folding, so incremental refresh won't work as expected
Is it not possible to check the files using the Modified Date? Similar to what was demonstrated here:
Faster SharePoint folder consolidation using Incremental Refresh (see warning in the notes)
v-ssriganeshCommunity Support
Hi olimilo,
Thank you for posting your query in the Microsoft Fabric Community Forum.Incremental refresh policies are configured at the individual table/query level and do not automatically propagate to referenced or merged queries. Therefore, you do not need to duplicate AuditWF, but if AUFI_1 and AUFI_2 are also loaded as separate tables and require incremental refresh behavior, you'll need to configure an incremental refresh policy for each of those tables separately.
Regarding your question about using SharePoint file metadata such as Modified Date, filtering based on metadata can help reduce the amount of data processed, depending on how the query is evaluated. However, incremental refresh relies on query folding to push filters to the data source. If the source does not support query folding, filtering might not be applied at the source level, which can reduce the effectiveness of incremental refresh.
In general, incremental refresh delivers the best performance when query folding is available, allowing filters to be applied at the source.
For more information, please refer to the official documentation:
Using incremental refresh with dataflows - Power Query | Microsoft Learn
Incremental refresh in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
Thanks, Kedar_Pande for sharing valuable insights.
Best regards,
Ganesh Singamshetty- v-ssriganesh
Hello olimilo,
We hope you're doing well. Could you please confirm whether your issue has been resolved or if you're still facing challenges? Your update will be valuable to the community and may assist others with similar concerns.
Thank you.
- v-ssriganesh
Hello olimilo,
Hope everything’s going great with you. Just checking, the issue been resolved or are you still running into problems? Sharing an update can really help others facing the same thing.
Thank you.