Forum Discussion
Upsert data to Lakehouse Table using copy data
Hi Alaahady ,
You need to transfer table from on-premises sql server to datalake using incremental update (upsert) . I have provided all the possible work arounds. But you are expecting Low code / No code. You can try the "Copy job" in Data Factory, makes it easy to move data from your source to your destination without creating a pipeline.
Please refer below links.
How to create a Copy job in Data Factory - Microsoft Fabric | Microsoft Learn
I hope this information helps. Please do let us know if you have any further queries.
Regards,
Dinesh
SQL Server On premises has no incremental refresh option. Move to Spark dataframe ( Data Lake ) and Delta Lake ( Data Lakehouse ) using cloud and it doesn't even require for incremental refresh. ( overwrite ). There is an incremental refresh in Spark but it requires large tables. ( millions of rows ).
# Notebooks
# Writing a DataFrame to Delta format
sdf.write.format("delta").option("overwrite").saveAsTable("DimTable")