topic read from azure storage container folder in Pipelines

read from azure storage container folder

Anonymous — Fri, 10 May 2024 08:12:17 GMT

hello,

I have managed to get the .csv file from the web page and place it into the azure storage container using the copy activity in fabric data factory.

Now I would like to read that .csv file, format some of the columns and then append this into a warehouse table (I created this table already)

any suggestions please?

thanks

Re: read from azure storage container folder

v-cboorla-msft — Fri, 10 May 2024 09:16:40 GMT

Hi Anonymous

Thanks for using Microsoft Fabric Community.

You can read the .csv file from azure storage container and append it to Fabric Warehouse by using Copy activity in Fabric Pipeline.

I hope this information helps. Please do let us know if you have any further queries.

Thank you.

Re: read from azure storage container folder

Anonymous — Fri, 10 May 2024 10:13:15 GMT

I see...

In that case, question please:

1- in the destination tab - pre-script - before data is inserted into a warehouse table, get the filename (I know I can get the filename at source copy activity using $$FILENAME but not sure how to pass it into the pre-script box in destination tab.

thanks

Re: read from azure storage container folder

v-cboorla-msft — Fri, 10 May 2024 10:42:37 GMT

Hi Anonymous

I would to like to understand what are you trying to achieve with the help of pre-script? If you are looking to adding the additional column value to warehouse, then you simply continue with mapping option.

If your query is different, please share more details of your query for better understanding.

Thank you.

Re: read from azure storage container folder

Anonymous — Fri, 10 May 2024 10:52:47 GMT

basically there is a table in the warehouse I created called tblwh The ADF pipeline reads the csv and places the data including the additional columnsi I added in th ecopy activity source nto this table tblwh. All good

Now if I re-run the pipeline I do not want the data to be duplicated into the tblwh. So I want to make sure it is deleted if it is already inserted in the previous pipeline run for the same filename.

and so I would like to have in the pre-scrip box to have something like:

delete from tblwh where filename = 'xyz'

Re: read from azure storage container folder

v-cboorla-msft — Mon, 13 May 2024 10:47:16 GMT

Hi Anonymous

I was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

I can find you were able to get a resolution in this thread : Solved: copy activity - pre-copy script - Microsoft Fabric Community

Thanks.

Re: read from azure storage container folder

v-cboorla-msft — Tue, 14 May 2024 10:07:47 GMT

Hi Anonymous

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks.

Re: read from azure storage container folder

Anonymous — Tue, 14 May 2024 10:11:08 GMT

all good now

using the presqlscript does the job

thank you