Number of partitions created when I output a parquet file

Anonymous — Fri, 19 May 2023 20:51:56 GMT

How can I control the number of partitions created when I output a parquet file?

Re: Number of partitions created when I output a parquet file

chetnachaudhari — Fri, 11 Aug 2023 00:46:00 GMT

Hi Anonymous,

If you are using PySpark, you can control the number of partitions created when you output a Parquet file by using the repartition method or the coalesce method on your DataFrame before writing it to Parquet. These methods allow you to control the number of output partitions, which in turn affects the number of Parquet files generated.

Thanks,

Chetna

topic Re: Number of partitions created when I output a parquet file in Data Warehouse

Number of partitions created when I output a parquet file

Re: Number of partitions created when I output a parquet file