Forum Discussion

Mathan1787's avatar
Mathan1787
Icon for Helper I rankHelper I
3 years ago

Upsert in copy activity

Hi,   could you let me know how can i do upsert  in copy activity of Data pipeline so i could load the data from Azure sql table to lakehouse and store the information as delta table or if there ar...
pipelines
Tips and Tricks
Mathan1787's avatar
Mathan1787
Icon for Helper I rankHelper I
3 years ago

I am planning to sort the upsert with notebook as copying the entire 2 billion records would be very costly. it would be nice if there is a incremental loading of data or cdc.

DennesTorres's avatar
DennesTorres
Icon for Power Participant rankPower Participant
3 years ago

Hi,

If you manage to identify an "updatedDate" and/or "InsertedDate" and/or an ascending field, such as Date or ascending numeric key, you can manage to build an incremental load based on them and still keep an image of the records you load every day.

Keeping this image is important, if you really can't find other solution than the upsert, you miss this image. But yes, I understand for this amount, if the incremental load is not possible, at least the upsert maybe needed.

Kind Regards,

Dennes