Forum Discussion
Hamidr
5 months agoFrequent Visitor
Upsert to Warehouse using copy activity vs notebook
Is there any differences in terms of performance between following two scenarios : 1- Running a pipeline every 15 minutes to read data from a lakehouse table and write into a warehouse table with u...
- 5 months ago
Hi Hamidr,
this really depends on the type of workload and how complex your logic is.
Both options will get the job done, but they behave a bit differently in practice.
Using Copy Activity (Upsert)
- This approach relies on Fabric’s built-in data movement engine, which is generally optimized for performance.
- It works best when your logic is straightforward, such as a simple key-based upsert using a column like EventID.
- Since it has less execution overhead, it’s usually a good fit for frequent runs (like every 15 minutes).
Using Notebook with MERGE
- A notebook gives you more flexibility, especially if your upsert logic involves additional conditions or transformations.
- You can fully control how records are inserted or updated using SQL MERGE.
- That said, it may introduce a bit more overhead due to compute and execution setup.
Thanks,
prashanth
v-prasare
Community Support
5 months agoHi Hamidr,
this really depends on the type of workload and how complex your logic is.
Both options will get the job done, but they behave a bit differently in practice.
Using Copy Activity (Upsert)
- This approach relies on Fabric’s built-in data movement engine, which is generally optimized for performance.
- It works best when your logic is straightforward, such as a simple key-based upsert using a column like EventID.
- Since it has less execution overhead, it’s usually a good fit for frequent runs (like every 15 minutes).
Using Notebook with MERGE
- A notebook gives you more flexibility, especially if your upsert logic involves additional conditions or transformations.
- You can fully control how records are inserted or updated using SQL MERGE.
- That said, it may introduce a bit more overhead due to compute and execution setup.
Thanks,
prashanth