Forum Discussion

smpa01's avatar
smpa01
Community Champion
2 years ago
Solved

Seeking partition strategy

I am in early phase of ingestion-building and delta table creation. I know that having partition even increases the performance. Therefore, I am seeking a partitioning strategy on the delta tables. ...
data engineering
lakehouse
notebook
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi smpa01 ,

     

    Thanks for the reply from frithjof_v .

     

    To verify partitions on a managed delta table, there are the following methods:

     

    1. Delta Lake stores partitioned data in a nested catalog structure. You can navigate to where the table is stored and examine the catalog structure to view the partitions.

     

    2. Check for the existence of partitions by using a SQL query.

     

    SELECT DISTINCT emp_id FROM people

     

     

    3. Use the Delta Lake API to check for partitions.

     

    from delta.tables import DeltaTable

delta_table = DeltaTable.forName(spark, "people")
delta_table.toDF().select("emp_id ").distinct().show()

     

     

    If you have any other questions please feel free to contact me.

     

    Best Regards,
    Yang
    Community Support Team

     

    If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
    If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

smpa01's avatar
smpa01
Community Champion
2 years ago

This article is not relatable to this question.

lbendlin's avatar
lbendlin
Super User
2 years ago

Delta Lake table optimization and V-Order - Microsoft Fabric | Microsoft Learn

Best practices — Delta Lake Documentation  

 

Choose the right partition column

You can partition a Delta table by a column. The most commonly used partition column is date. Follow these two rules of thumb for deciding on what column to partition by:

  • If the cardinality of a column will be very high, do not use that column for partitioning. For example, if you partition by a column userId and if there can be 1M distinct user IDs, then that is a bad partitioning strategy.

  • Amount of data in each partition: You can partition by a column if you expect data in that partition to be at least 1 GB.