Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.
How can I control the number of partitions created when I output a parquet file?
Solved! Go to Solution.
Hi @saveenrMSFT,
If you are using PySpark, you can control the number of partitions created when you output a Parquet file by using the repartition method or the coalesce method on your DataFrame before writing it to Parquet. These methods allow you to control the number of output partitions, which in turn affects the number of Parquet files generated.
Thanks,
Chetna
Hi @saveenrMSFT,
If you are using PySpark, you can control the number of partitions created when you output a Parquet file by using the repartition method or the coalesce method on your DataFrame before writing it to Parquet. These methods allow you to control the number of output partitions, which in turn affects the number of Parquet files generated.
Thanks,
Chetna
Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City
Check out the March 2024 Fabric update to learn about new features.
User | Count |
---|---|
7 | |
6 | |
5 | |
4 | |
3 |