Forum Discussion

Hamidr's avatar
Hamidr
Frequent Visitor
5 months ago
Solved

Upsert to Warehouse using copy activity vs notebook

Is there any differences in terms of performance between following two scenarios :  1- Running a pipeline every 15 minutes to read data from a lakehouse table and write into a warehouse table with u...
  • v-prasare's avatar
    5 months ago

    Hi Hamidr,

    this really depends on the type of workload and how complex your logic is.

    Both options will get the job done, but they behave a bit differently in practice.

     Using Copy Activity (Upsert)

    • This approach relies on Fabric’s built-in data movement engine, which is generally optimized for performance.
    • It works best when your logic is straightforward, such as a simple key-based upsert using a column like  EventID.
    • Since it has less execution overhead, it’s usually a good fit for frequent runs (like every 15 minutes).

    Using Notebook with MERGE

    • A notebook gives you more flexibility, especially if your upsert logic involves additional conditions or transformations.
    • You can fully control how records are inserted or updated using SQL MERGE.
    • That said, it may introduce a bit more overhead due to compute and execution setup.

     

     

    Thanks,
    prashanth