topic Re: connecting dataflow to a lakehouse folder of files to combine them in Dataflow

connecting dataflow to a lakehouse folder of files to combine them

Anonymous — Sat, 18 May 2024 21:24:20 GMT

I want to use dataflow gen2 to process a folder full of CSV files (all identical in structure) and load the combined and cleaned data in lakehouse table. The folder is in a lakehouse in the same premium workspace as the dataflow.

miguel in Solved: Re: Can Dataflow Gen2 read from lakehouse "Files" ... - Microsoft Fabric Community says "You can use the combine files experience the same as with other file system views (SharePoint, data lake, folder and others). " Therefore I'm expecting to see a "Combine" button when i connect to the folder, but I do not. Am I doing something wrong?

Details.

1. New dataflow gen2
2. Get data> More
3. OneLake data hub
4. Click on the lakehouse
5. browse to the folder. Note I can select files individual, or stay on the folder name. The only option for the folder is "Create", as shown:

If I choose Create I get a table with a list of the file contents, as shown:

Once I have that, I can drill down into one row at at time to get the csv file, but not sure how to process all of them efficiently (there will be dozens in production).

Any advice, other than using a notebook instead .

Re: connecting dataflow to a lakehouse folder of files to combine them

v-cboorla-msft — Mon, 20 May 2024 17:35:22 GMT

Hi Anonymous

Thanks for using Microsoft Fabric Community.

I tried to repro the scenario and able to find the "combine" option in Dataflow Gen2, this happens only when I am loading data from external storage (in my case I used ADLS Gen2). Please find the screenshot for reference.

I hope this information helps. Please do let us know if you have any further queries.

Thank you.

Re: connecting dataflow to a lakehouse folder of files to combine them

Anonymous — Mon, 20 May 2024 18:33:47 GMT

If I connect via Azure Gen 2 storage then yes I see the Combine button. However, not when connecting via the Onelake shortcut (as described above): can you try that please? Connecting via storage requires me to get first the https://onelake.dfs.fabric.microsoft.com/ URL, sign in again, and I think it also requires On-premise data gateway.

Re: connecting dataflow to a lakehouse folder of files to combine them

brackley2 — Thu, 06 Nov 2025 05:45:20 GMT

Definitely having the same experience...so frustrating. Why is the combine files option disabled when connecting to onelake?

I loaded the same files to a sharepoint folder and connected via DFGen2. The combine option appeared as I would've expected.