Forum Discussion

DennesTorres's avatar
DennesTorres
Icon for Power Participant rankPower Participant
3 years ago

PARQUET file size and updates

Hi,   Usually we try to keep the parquet file sizes large, otherwise the excess of small files can create problems for processing. The default file size, unless I'm mistaken, is set for 1GB. Most e...
data engineering
HimanshuS-msft's avatar
HimanshuS-msft
Icon for Microsoft Employee rankMicrosoft Employee
3 years ago

Hello DennesTorres 

I think this is catch 22 situation , if you keep the file size at 1 GB updates are slow but read is fine m otherwise updates are fine and read is slow . As i understand from the post that you do not perform a lot of updates and do perform lot of read and so I think going ahead with 1 GB file size , should be the right approah .

 

Thanks 
Himanshu 

 

 

DennesTorres's avatar
DennesTorres
Icon for Power Participant rankPower Participant
3 years ago

Hi,

 

Thank you, I also agree and It's important to get this kind of feedback.

 

I believe as a result, when someone decides to use an upsert in opposited to a dimension type 2 (SCD), this has a consequence and they should know that to take a decision if 1GB of file size is good for their solution. Or not? Any thoughts about this?

I'm also wondering why we have no control about this on the data warehouse, I posted a different question about this.

 

Kind Regards,

 

Dennes