topic Re: Data pipeline to download zip file containing multiple csv files from web in Pipelines

Data pipeline to download zip file containing multiple csv files from web

Anonymous — Fri, 02 Feb 2024 20:54:59 GMT

I am attempting to create a pipeline or pipelines to do the following:

Download a zip file from a website (https://www150.statcan.gc.ca/n1/en/tbl/csv/14100287-eng.zip?st=0Sbmw1wJ)
Store the binary zip to a lakehouse in the Files/Zip
Unzip the downloaded file into multiple CSV files in Files/CSV (folder in the lakehouse)
Dataflow Gen2 to perform the ETL functions

Re: Data pipeline to download zip file containing multiple csv files from web

Anonymous — Fri, 02 Feb 2024 20:56:43 GMT

I am attempting to create a pipeline or pipelines to do the following:

Download a zip file from a website (https://www150.statcan.gc.ca/n1/en/tbl/csv/14100287-eng.zip?st=0Sbmw1wJ)
Store the binary zip to a lakehouse in a folder - Files/Zip
Unzip the downloaded file into multiple CSV files in Files/CSV (folder in the lakehouse)
Dataflow Gen2 to perform the ETL functions to create a table in the Data lake

The problem is that when I unzip the file into delimited format, there is no way to unzip multiple files to the Files/CSV folder. The get data in the pipeline wants to create a single file. I have attempted to go directly from binary zip to table, but the procedure gives me an error because the two CSV are not in the same format.

I am looking for help on the unzip pipeline into multiple CSV files.

Re: Data pipeline to download zip file containing multiple csv files from web

Brendan151 — Mon, 05 Feb 2024 00:50:31 GMT

You should be able to do this in a Gen2 Dataflow, following tackytechtom's answer from https://community.fabric.microsoft.com/t5/Power-Query/zipped-file-how-to-unzip-in-power-query-more-explanations-needed/td-p/2949359

Run the function with your url and then you can create separate queries in Power Query for each CSV file.

Re: Data pipeline to download zip file containing multiple csv files from web

v-cboorla-msft — Wed, 07 Feb 2024 11:03:22 GMT

Hi Anonymous

Thanks.

Re: Data pipeline to download zip file containing multiple csv files from web

v-cboorla-msft — Thu, 08 Feb 2024 06:11:33 GMT

Hi Anonymous

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread.

Thanks.

Re: Data pipeline to download zip file containing multiple csv files from web

Anonymous — Wed, 14 Feb 2024 22:56:22 GMT

I would really like a data pipeline solution to this. It seems like the ability to unzip delimitated files is built-in, but it fails due to the lack of proper handling of multiple files in the zip. I would love to see a pipeline procedure that unzips multiple files to the Files folders. Other pipelines would process the csv and combine them. Any chance that notebooks could solve my problem?

Re: Data pipeline to download zip file containing multiple csv files from web

Ostrzak — Wed, 16 Oct 2024 11:39:41 GMT

Upvoting the question. I have a similar problem, where pipeline is extracting the csv file, but there are some parsing errors.