topic Re: Copy a file from an URL to OneLake in Pipelines

Copy a file from an URL to OneLake

mrojze — Tue, 27 Feb 2024 21:47:30 GMT

I have a Fabric pipeline that is requesting an Amazon SP report through API calls. Amazon prepares the report and it returns a unique URL to download the report.

How can I download the report using DF?

A WEB call requires a base URL, which I don't have.

Any suggestions?

Thanks!

Martin Rojze

Re: Copy a file from an URL to OneLake

Anonymous — Wed, 28 Feb 2024 09:31:22 GMT

Hi mrojze ,

Thanks for using Fabric Community.
You can only save the dataframe to file, but you cannot download a file while working with notebooks.

If you are passing url from notebook activity to notebook using fabric data factory.

After reading the notebook parameters value, you can make a requests get call but gets the data in string format only.

Hope this is helpful. Please let me know incase of further queries.

Re: Copy a file from an URL to OneLake

mrojze — Wed, 28 Feb 2024 13:56:00 GMT

Thanks for your reply!

So what are my options if I have a URL with a GZIP?

Is there any way to download it?

Maybe using LogicApps?

Re: Copy a file from an URL to OneLake

Anonymous — Thu, 29 Feb 2024 07:05:01 GMT

Hi mrojze ,

I found few similar threads on this topic -
How do I bring data using URL (with Azure Logic App)? - Microsoft Q&A
Download file from HTTPS link received in email using Logic Apps Step by Step Process - Microsoft Q&A

Hope this might be helpful.

Re: Copy a file from an URL to OneLake

Anonymous — Fri, 01 Mar 2024 07:09:03 GMT

Hello mrojze ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .

Re: Copy a file from an URL to OneLake

mrojze — Fri, 01 Mar 2024 22:18:28 GMT

Yes! I have solved the issue using my new adquired Python skills (aka Copilot).

In order to download the file I do the following:

I capture the URL in the vairable url and then I ised the described library in order to use the urlretrieve method.

The issue I found after it was that the downloaded file was a GZIP one. Guess what?

A few more libraries and a few lines of code later I was able to place the unzipped file into the OneLake:

Thanks anyone that responded to my post.

Re: Copy a file from an URL to OneLake

SyedN — Tue, 30 Jul 2024 02:20:34 GMT

mrojze I am looking to ingest JSON file in my android download folder and tryig to push it through my android APP ( using Java ) to write it on One Lake can you please guide me how to do this ?