Merge multiple files into one file in a lakehouse.

[esgaranel]

Hello, could someone tell me how to do this process?

Thanks

[esgaranel]

Thank you so much

[suparnababu8 · Accepted solution]

Hello @esgaranel 

Based on your archetecture, I suggest two ways.

1) Using DataflowGen2  (Low-code tool)

• You can load  the data from you database/local into Dataflow Gen2 using pipeline(if it's in database) / directly uploading (from local).
• Once you got the your files into Dataflowgen2, you can do the merge by using merge option in dataflowgen2.
• Once megre was completed, later you can connect you respective lakehouse and push the data to lakehouse.

Please go through this links for this step1: https://learn.microsoft.com/en-us/fabric/data-factory/create-first-dataflow-gen2, https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-gen2-data-destinations-and-managed-se... , 

2) Using Notebooks in Fabric environment. 

• You can create a lakehouse and upload files into lakehouse (dirfect upload/ use shortcuts). the below are supported shortcuts in fabric
   
• Later you can connect this lakehouse with notebook and merge all you files into single file. 
• Then once merging completed, you can write to lakehouse as delta tbale

Please go through this link for step 2 : https://community.fabric.microsoft.com/t5/Desktop/Combine-specific-Files-from-a-Folder/td-p/3546941 

Please ket me know if it works.

Tahnk you!!

Did I answer your question? Mark my post as a solution!

Proud to be a Super User!