Forum Discussion

olimilo's avatar
olimilo
Icon for Post Prodigy rankPost Prodigy
5 months ago
Solved

Does incremental refresh cascade down to referenced/merged queries?

We're currently loading 6 or so years worth of data from Excel files in Sharepoint into the AuditWF table. The output from that query is then referenced and merged with 2 separate tables, resulting i...
  • Kedar_Pande's avatar
    5 months ago

    olimilo 

     

    Incremental refresh does not cascade. You need to configure it separately on AuditWF, AUFI_1, and AUFI_2.
    One thing to flag: SharePoint Excel files don't support query folding, so incremental refresh won't work as expected. It will pull the full data every time. Move to a SQL or Lakehouse source if you want incremental refresh to actually work.

  • v-ssriganesh's avatar
    5 months ago

    Hi olimilo,
    Thank you for posting your query in the Microsoft Fabric Community Forum.

    Incremental refresh policies are configured at the individual table/query level and do not automatically propagate to referenced or merged queries. Therefore, you do not need to duplicate AuditWF, but if AUFI_1 and AUFI_2 are also loaded as separate tables and require incremental refresh behavior, you'll need to configure an incremental refresh policy for each of those tables separately.

    Regarding your question about using SharePoint file metadata such as Modified Date, filtering based on metadata can help reduce the amount of data processed, depending on how the query is evaluated. However, incremental refresh relies on query folding to push filters to the data source. If the source does not support query folding, filtering might not be applied at the source level, which can reduce the effectiveness of incremental refresh.

    In general, incremental refresh delivers the best performance when query folding is available, allowing filters to be applied at the source.

    For more information, please refer to the official documentation:
    Using incremental refresh with dataflows - Power Query | Microsoft Learn
    Incremental refresh in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

    Thanks, Kedar_Pande for sharing valuable insights.
    Best regards,
    Ganesh Singamshetty