topic Re: Copy data from Folder in Data Warehouse

Copy data from Folder

amymc — Tue, 30 Jul 2024 08:57:35 GMT

Hi,

Im trying to use a data pipeline in Fabric to copy data from a folder into my lakehouse. However when I put in my folder address the button for next is greyed out. When I click the link for more help it says that Folder is only available from Dataflow Gen2 for Fabric. I am testong it with a local folder for now, but will eventually be using a network folder. Just wondering will i have to use DataFlow gen2 for this.

Thanks

Re: Copy data from Folder

AndyDDC — Tue, 30 Jul 2024 13:01:19 GMT

HI amymc where is the local folder you're trying to copy from?

Re: Copy data from Folder

amymc — Tue, 30 Jul 2024 13:04:04 GMT

I tried from oneDrive (C:\Users\amy\OneDrive\Documents\Data) and from local ( C:\Users\amy\Documents\Data)

Re: Copy data from Folder

AndyDDC — Tue, 30 Jul 2024 13:13:32 GMT

Do you have an "on-premises data gateway" installed on the computer where you're trying to access the local folder? In order to load data from a local computer into Fabric, you need a gateway in order for Fabric it to access it

Re: Copy data from Folder

amymc — Tue, 30 Jul 2024 13:21:32 GMT

No, but I am able to access the data from dataflow gen2 without an on-premise data gateway, why is this possible.

Thank you

Re: Copy data from Folder

AndyDDC — Tue, 30 Jul 2024 13:36:57 GMT

How are you accessing the data using Dataflow Gen2? If it's Onedrive then the data will be synced from your local PC to cloud onedrive, which is what dataflow gen2 will then access.

It is not physically possible for Fabric/Power BI to access folders/files on your local PC without a gateway installed.

btw Data Pipelines do not support connecting to cloud onedrive.

Re: Copy data from Folder

Anonymous — Wed, 31 Jul 2024 02:39:18 GMT

Hi amymc ,

Thanks for the reply from AndyDDC , his answer works like a charm!

Your problem should be in the gateway.

I used pipeline for copy data to lakehouse and it worked, here are my test steps:

First, select Folder for the Get Data method, then configure it, note that you must have a gateway (because the folder is local) and the gateway must be running, select the appropriate authentication method.

Just choose a lakehouse as your destination.

Then you can click on the Validate and if there are no problems you can Run it.

After running successfully, the effect is as shown below:

You mentioned that you were able to access data from dataflow gen2 without a local data gateway, I'm a bit confused because when I test it if I want to access the local data it reminds me to configure the gateway, like in the picture below:

I would like to know how you are able to access the local Folder without a gateway?

Please configure the gateway and run it and then try again to connect to the local folder using pipeline.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

Re: Copy data from Folder

amymc — Wed, 31 Jul 2024 07:45:51 GMT

Hi,

Thanks for your reply. I checked again and it was a OneDrive folder I was actually acessing from Dataflow. I wasnt able to access this folder from the pipeline through but am from Dataflows.

Your answer and steps are super helpful. Just wondering will I need this a gateway be able to access network folders too.

Thank you

Re: Copy data from Folder

AndyDDC — Wed, 31 Jul 2024 08:05:40 GMT

Is there a reason my answer wasn't accepted as the solution?