Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
Anonymous
Not applicable

Number of partitions created when I output a parquet file

How can I control the number of partitions created when I output a parquet file?

1 ACCEPTED SOLUTION
chetnachaudhari
Advocate I
Advocate I

Hi @Anonymous,

  If you are using PySpark, you can control the number of partitions created when you output a Parquet file by using the repartition method or the coalesce method on your DataFrame before writing it to Parquet. These methods allow you to control the number of output partitions, which in turn affects the number of Parquet files generated.

Thanks,

Chetna

View solution in original post

1 REPLY 1
chetnachaudhari
Advocate I
Advocate I

Hi @Anonymous,

  If you are using PySpark, you can control the number of partitions created when you output a Parquet file by using the repartition method or the coalesce method on your DataFrame before writing it to Parquet. These methods allow you to control the number of output partitions, which in turn affects the number of Parquet files generated.

Thanks,

Chetna

Helpful resources

Announcements
Sept Fabric Carousel

Fabric Monthly Update - September 2024

Check out the September 2024 Fabric update to learn about new features.

September Hackathon Carousel

Microsoft Fabric & AI Learning Hackathon

Learn from experts, get hands-on experience, and win awesome prizes.