Forum Discussion
upsert option missing in copy data activity
in this documentation :
Load data to Lakehouse using partition - Microsoft Fabric | Microsoft Learn
we can see an upsert option in the screenshots to load data in a lakehouse.
but I'm not able to find it on my side. (with lakehouse or warehouse destination)
is the upsert feature supposed to be available?
if no, what are the alternatives to do it?
for now it's pretty complex to update data in a lakehouse. So for now I'm using the overwrite option, but this causes issues to my users if they are opening a report in the middle of a loading of the data.
Hello Jerome22 ,
As called Anonymous , at this time the UPSERT is not avalable . Since you mentioned
"for now it's pretty complex to update data in a lakehouse. So for now I'm using the overwrite option, but this causes issues to my users if they are opening a report in the middle of a loading of the data."
I am assuming that the you are using Direct query for the reports and request you to see you can use the Import option , this is the link : https://learn.microsoft.com/en-us/power-bi/connect-data/desktop-use-directquery
Thanks
Himanshu
8 Replies
Hi Jerome22
Thanks for using Fabric Community.
Apologies for the issue you have been facing. The "Upsert option is not available yet". This feature is on roadmap.
You can also try using Dataflow Gen2 for an incremental load. For more information please refer to this document: https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-dataflows-gen2
Hope this helps. Please let me know if you have any further questions.Hi Jerome22
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Otherwise, will respond back with the more details and we will try to help.
ThanksHi @Jerome22
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Otherwise, will respond back with the more details and we will try to help.
Thanks
Hi Anonymous ,
Just wondering - do you think the upsert option will become available in Fabric data pipelines when the T-SQL Merge feature is released?
MERGE (T-SQL)
Estimated release timeline: Q1 2025Release Type: General availability
The MERGE T-SQL command for Fabric Data Warehouse brings the power of selection-based DML into a single statement, for all your transformation logic needs.
At the moment I am receiving gzipped json files and I basically need to copy data into staging tables, before using Notebook to achieve the upsert/merge. This would be a game changer for me as the copy data activity is much quicker than starting multiple notebooks in a for each loop.
Thank you,
Josh
Hello Jerome22 ,
As called Anonymous , at this time the UPSERT is not avalable . Since you mentioned
"for now it's pretty complex to update data in a lakehouse. So for now I'm using the overwrite option, but this causes issues to my users if they are opening a report in the middle of a loading of the data."
I am assuming that the you are using Direct query for the reports and request you to see you can use the Import option , this is the link : https://learn.microsoft.com/en-us/power-bi/connect-data/desktop-use-directquery
Thanks
HimanshuUnfortunatly I don't want to import the data.
the entire idea is to not duplicate the data and not over processing the data.
I will do differently for now, but any idea when the upsert will be available?
Hi Jerome22
Did you check this link: https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-dataflows-gen2?
This is equivalent to upsert condition.
