Forum Discussion
How does Destination > Overwrite with a partition columns work ?
- 2 years ago
gregbortolotti - Overwrite with a Partition specific will overwrite the entire partition, it does not perform a merge. Likewise, the Append will add new rows the partition but will not update the old versions. You can do a merge however in Notebooks, we do a lot of that with Partitioned and Non-Partitioned data, link to the Databricks documentation (still applies to Fabric, I just like the documentation better).
Partitioning for very large tables to help speed up data access activities based on the partition field. Think of it as grouping data/transactions together. If you have transactional data that doesn't change, sometimes you would partition that by Transaction Date; that way when looking for Transactions in that time period it only queries that Partition vs the whole dataset.
- 2 years ago
Some more options for how to update or upsert by using the Delta Lake Python API in a Notebook:
Thank you very much for explanation. Now it is clearer to me now.
Best regards
Here is also a Fabric Idea to get the Upsert functionality natively in Data Pipeline.
Consider voting to highlight this need:
Support UPSERTs and DELETEs when copying data into Lakehouse Tables from Pipeline copy activity, as opposed to Appending new rows
https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=87f3d168-6022-ee11-a81c-6045bdc01ce4