Upsert to Warehouse using copy activity vs notebook

[Hamidr]

Is there any differences in terms of performance between following two scenarios : 
1- Running a pipeline every 15 minutes to read data from a lakehouse table and write into a warehouse table with using upsert write method with using copy data activity (there is a column "EventID" to use for upsert)

2- Every 15 minutes trigger a notebook which is doing same upsert action with using merge command.

Thanks

[tayloramy]

Hi @Hamidr, 

Generally speaking, the less code you write the more expensive the operation. 

Notebooks will be more efficient than a copy activity in an ideal world, but that depends on writing efficient code and having a properly sized spark cluster, or for small workloads dropping spark entirely and using pure python notebooks. 

If you're skilled at python, then use a notebook. If you're not, stick with the pipeline and copy activity. 

[v-prasare · Accepted solution]

Hi @Hamidr,

this really depends on the type of workload and how complex your logic is.

Both options will get the job done, but they behave a bit differently in practice.

 Using Copy Activity (Upsert)

• This approach relies on Fabric’s built-in data movement engine, which is generally optimized for performance.
• It works best when your logic is straightforward, such as a simple key-based upsert using a column like  EventID.
• Since it has less execution overhead, it’s usually a good fit for frequent runs (like every 15 minutes).

Using Notebook with MERGE

• A notebook gives you more flexibility, especially if your upsert logic involves additional conditions or transformations.
• You can fully control how records are inserted or updated using SQL MERGE.
• That said, it may introduce a bit more overhead due to compute and execution setup.

Thanks,
prashanth