Forum Discussion
partitioning concept
Hi Jeanxyz ,
Yes, you’re absolutely right. If your Delta table is partitioned by department but your merge condition is based on employee_id, then the partitioning doesn’t really help during the merge. Spark will still need to scan all partitions to find the matching employee_id values, because the partition column isn’t part of the predicate.
Partitioning only speeds things up when your filter, join, or merge condition includes the partition column itself. For example, a merge or query that filters on department = 'HR' would benefit immediately, but a merge on employee_id won’t take advantage of department partitions.
So the general rule is:
Pick a partition column only if your downstream workloads actually filter or process data using that column. Otherwise, partitioning doesn’t provide much benefit.Hope this helps — and glad the earlier correction was useful. If this answers your question, please mark it as Accepted Solution ✔️
– Gopi Krishna
Hi Jeanxyz ,
Your understanding is close, but there’s an important detail to clarify. Partitioning does work when writing Delta tables — it doesn’t only apply to writing files. The issue you hit is just the syntax.
In Spark, the partitionBy() method is always part of the writer, not an argument inside saveAsTable. So your corrected syntax is the right one. When you write:
df.write \
.mode("overwrite") \
.format("delta") \
.partitionBy("department") \
.saveAsTable("data_with_partition")
Spark will create a fully valid partitioned Delta table, and this works perfectly fine with Power BI, Fabric Lakehouse, and SQL endpoint queries.
So partitioning is still useful even if your final destination is a Delta table. As long as your table has a natural grouping column with reasonable cardinality (like department, date, region, category, etc.), partitioning can improve performance for large datasets.
Your updated example using:
saveAsTable("data_withoutpart2", partitionBy="department")
is also valid — it’s just an alternative syntax that Spark supports.
So yes, your data can be written as Delta and still be partitioned without any problem. You weren’t running into a conceptual limitation, just a small syntax detail.
Hope this helps. If it does, please give a Kudos 👍 or mark as Accepted Solution ✔️
– Gopi Krishna
Thanks for clarification, Ugk161610 .
One last question: if my delta table is partitioned by 'department' but I'm running a merge query by 'employee_id', the partition won't help in the case. Is that correct?
Hi Jeanxyz ,
Yes, you’re absolutely right. If your Delta table is partitioned by department but your merge condition is based on employee_id, then the partitioning doesn’t really help during the merge. Spark will still need to scan all partitions to find the matching employee_id values, because the partition column isn’t part of the predicate.
Partitioning only speeds things up when your filter, join, or merge condition includes the partition column itself. For example, a merge or query that filters on department = 'HR' would benefit immediately, but a merge on employee_id won’t take advantage of department partitions.
So the general rule is:
Pick a partition column only if your downstream workloads actually filter or process data using that column. Otherwise, partitioning doesn’t provide much benefit.Hope this helps — and glad the earlier correction was useful. If this answers your question, please mark it as Accepted Solution ✔️
– Gopi Krishna