Forum Discussion
Incremental Refresh with Appended Web Source
Hi aviha12 ,
Thank you for reaching out to Microsoft Fabric Community.
Thank you GilbertQ lbendlin for the prompt response.
To make incremental refresh work when appending SQL + non-folding Web data, separate the queries.Apply incremental refresh only on the SQL fact tables (they fold). Load the Web tables separately as small “today-only” datasets without incremental refresh.Then create a final table that simply UNIONs/APPENDs the SQL incremental table with the Web table.The incremental partitions are created from the SQL source, and the Web data is small enough to load fully each refresh.This pattern avoids breaking query folding and is the Microsoft-recommended approach for combining folding + non-folding sources.
That's something you would need to do in code, for example via EXCEPT. Having duplicated data across partitions is very highly undesirable.
each table needs to be appended with data from a web source before loading to the model Does that web source have a filter option?
Regardless, since it doesn't fold you will not get any performance improvements with incremental refresh.
Rethink your approach entirely. Push both the web data and the SQL Server data into CSV Zip archives on a Sharepoint. Use the file names to define the partition boundaries. Even though this doesn't fold it is still fast enough as the network transfer penalty is reduced.
I can't do it since the data is oragnizational data.
