Forum Discussion
Incremental Refresh with Appended Web Source
Hi aviha12 ,
Thank you for reaching out to Microsoft Fabric Community.
Thank you GilbertQ lbendlin for the prompt response.
To make incremental refresh work when appending SQL + non-folding Web data, separate the queries.Apply incremental refresh only on the SQL fact tables (they fold). Load the Web tables separately as small “today-only” datasets without incremental refresh.Then create a final table that simply UNIONs/APPENDs the SQL incremental table with the Web table.The incremental partitions are created from the SQL source, and the Web data is small enough to load fully each refresh.This pattern avoids breaking query folding and is the Microsoft-recommended approach for combining folding + non-folding sources.
That's something you would need to do in code, for example via EXCEPT. Having duplicated data across partitions is very highly undesirable.
Hi aviha12 ,
Thank you for reaching out to Microsoft Fabric Community.
Thank you GilbertQ lbendlin for the prompt response.
To make incremental refresh work when appending SQL + non-folding Web data, separate the queries.Apply incremental refresh only on the SQL fact tables (they fold). Load the Web tables separately as small “today-only” datasets without incremental refresh.Then create a final table that simply UNIONs/APPENDs the SQL incremental table with the Web table.The incremental partitions are created from the SQL source, and the Web data is small enough to load fully each refresh.This pattern avoids breaking query folding and is the Microsoft-recommended approach for combining folding + non-folding sources.
Thank you,
Will there be a way to delete duplicates on the final table?
Because in the Web table I also get rows that exist in the SQL table, but their status or whatever was updated. so if there is an ID that exist in both table, by remove duplicates it removes for me the one in the SQL table because of the order that the tables are appended, is it possible to do it in your solution?
And also, many of the columns that exist in the SQL table does NOT exist in the WEB table, will it still work?
Thanks!
That's something you would need to do in code, for example via EXCEPT. Having duplicated data across partitions is very highly undesirable.