upsert option missing in copy data activity

[Jerome22]

in this documentation :

Load data to Lakehouse using partition - Microsoft Fabric | Microsoft Learn

we can see an upsert option in the screenshots to load data in a lakehouse.

but I'm not able to find it on my side. (with lakehouse or warehouse destination)

is the upsert feature supposed to be available?

if no, what are the alternatives to do it?

for now it's pretty complex to update data in a lakehouse. So for now I'm using the overwrite option, but this causes issues to my users if they are opening a report in the middle of a loading of the data.

[HimanshuS-msft · Accepted solution]

Hello @Jerome22 ,

As called @Anonymous  , at this time the UPSERT is not avalable . Since you mentioned 
"for now it's pretty complex to update data in a lakehouse. So for now I'm using the overwrite option, but this causes issues to my users if they are opening a report in the middle of a loading of the data."
I am assuming that the you are using Direct query for the reports and request you to see you can use the Import option , this is the link : https://learn.microsoft.com/en-us/power-bi/connect-data/desktop-use-directquery
Thanks 
Himanshu

[Jerome22]

Unfortunatly I don't want to import the data.

the entire idea is to not duplicate the data and not over processing the data.

I will do differently for now, but any idea when the upsert will be available?

[Anonymous]

Hi @Jerome22 
Did you check this link: https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-datafl...
This is equivalent to upsert condition.

[Anonymous]

Hi @Jerome22 
Currently we do not have any update regarding the Upsert option availability in Copy data. But the incremental refresh in Dataflow Gen2 is on roadmap.

For more information please refer to this link:
https://learn.microsoft.com/en-us/fabric/release-plan/data-factory

Hope this helps. Please let me know if you have any further questions.

[Anonymous]

Hi @Jerome22 
Thanks for using Fabric Community.
Apologies for the issue you have been facing. The "Upsert option is not available yet". This feature is on roadmap.
You can also try using Dataflow Gen2 for an incremental load. For more information please refer to this document: https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-datafl...

Hope this helps. Please let me know if you have any further questions.

[efsjosh]

Hi @Anonymous ,

Just wondering - do you think the upsert option will become available in Fabric data pipelines when the T-SQL Merge feature is released?

MERGE (T-SQL)
Estimated release timeline: Q1 2025

Release Type: General availability

The MERGE T-SQL command for Fabric Data Warehouse brings the power of selection-based DML into a single statement, for all your transformation logic needs.

At the moment I am receiving gzipped json files and I basically need to copy data into staging tables, before using Notebook to achieve the upsert/merge. This would be a game changer for me as the copy data activity is much quicker than starting multiple notebooks in a for each loop. 

Thank you, 

Josh

[Anonymous]

Hi @Jerome22 
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Otherwise, will respond back with the more details and we will try to help.
Thanks

[Anonymous]

Hi @Jerome22 
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Otherwise, will respond back with the more details and we will try to help.
Thanks