Forum Discussion

waltheed's avatar
waltheed
Icon for Impactful Individual rankImpactful Individual
3 years ago
Solved

Fabric Data Factory Pipeline: Incremental load

How can I create an incremental load in Data Factory Pipeline?   I only see the Table Actions append or overwrite. But I want to be able to add new rows, update existing ones and mark deleted rows....
Need Help
pipelines
  • ajarora's avatar
    ajarora
    3 years ago

    Your table action options imply that you are using pipelines. If so, good news, we have just added a new feature that allows you to "upsert" the new rows into existing tables instead of append/overwrite only.

    Its in private preview, please ping me with your workspace id, and we can enable it for you along with instructions. cc makromer 

miguel's avatar
miguel
Icon for Community Admin rankCommunity Admin
3 years ago

Hey!

At the moment, we don't have an incremental refresh feature in the same way that Dataflows Gen1 have them in something like Power BI. We are internally tracking this and I've also created an idea so you can vote for it and help us prioritize it:

https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=4814b098-efff-ed11-a81c-6045bdb98602 

 

There are some alternatives or different approaches to incremental refresh depending on what logic your incremental refresh should follow by leveraging the existing capabilities with pipelines (for the orchestration component) and the output destinations using either an Append method or loading to a staging table and then running any sort of scripts that you'd like to run to make the incremental refresh happen through a notebook.