Dataflow - Combine files - performance

Hello all,

 

We have few hundred files of semi-structured data on ADLS Gen 2. Every thing is working fine. 

 

Dataflow is developed as below:

1. Get the files list
2. Combine all the files
3. * i.e., post files combining * Split into multiple tables. Each file will have 4 types of tables:
   1. Sales by Region
   2. Outstanding dues by Region
   3. Optional Table: Returns this period, optional table. If No returns, Prior to 2020, placeholder text was used saying, "no returns for this period" and from 2020, this table has become optional.
   4. Summary
4. Apply transformations - split, trim, clean, fill down, ... logic based on table.

 

Performance: It is taking time to process as we are doing transformations post file combining. Any recommendations to improve performance?

 

Best practices question - any advanced techniques to combine and transform these files?

 

Thanks in advance.