Forum Discussion
Incremental Refresh with Appended Web Source
Hi aviha12 ,
Thank you for reaching out to Microsoft Fabric Community.
Thank you GilbertQ lbendlin for the prompt response.
To make incremental refresh work when appending SQL + non-folding Web data, separate the queries.Apply incremental refresh only on the SQL fact tables (they fold). Load the Web tables separately as small “today-only” datasets without incremental refresh.Then create a final table that simply UNIONs/APPENDs the SQL incremental table with the Web table.The incremental partitions are created from the SQL source, and the Web data is small enough to load fully each refresh.This pattern avoids breaking query folding and is the Microsoft-recommended approach for combining folding + non-folding sources.
That's something you would need to do in code, for example via EXCEPT. Having duplicated data across partitions is very highly undesirable.
Hi aviha12
One alternative approach I would think of is if you have the incremental refresh being applied to your SQL table that will always then incrementally refresh the data and then in the subsequent steps just append your web source. That should work fine because your web source is always getting all the data, but automatically first apply the incremental refresh and then append the data.
But incremental refresh is being done in the DAX level. so I can't do it.
