Forum Discussion
How does Destination > Overwrite with a partition columns work ?
gregbortolotti - Overwrite with a Partition specific will overwrite the entire partition, it does not perform a merge. Likewise, the Append will add new rows the partition but will not update the old versions. You can do a merge however in Notebooks, we do a lot of that with Partitioned and Non-Partitioned data, link to the Databricks documentation (still applies to Fabric, I just like the documentation better).
Partitioning for very large tables to help speed up data access activities based on the partition field. Think of it as grouping data/transactions together. If you have transactional data that doesn't change, sometimes you would partition that by Transaction Date; that way when looking for Transactions in that time period it only queries that Partition vs the whole dataset.
Some more options for how to update or upsert by using the Delta Lake Python API in a Notebook:
gregbortolotti - Overwrite with a Partition specific will overwrite the entire partition, it does not perform a merge. Likewise, the Append will add new rows the partition but will not update the old versions. You can do a merge however in Notebooks, we do a lot of that with Partitioned and Non-Partitioned data, link to the Databricks documentation (still applies to Fabric, I just like the documentation better).
Partitioning for very large tables to help speed up data access activities based on the partition field. Think of it as grouping data/transactions together. If you have transactional data that doesn't change, sometimes you would partition that by Transaction Date; that way when looking for Transactions in that time period it only queries that Partition vs the whole dataset.
Some more options for how to update or upsert by using the Delta Lake Python API in a Notebook: