read from azure storage container folder

[arkiboys]

hello,

I have managed to get the .csv file from the web page and place it into the azure storage container using the copy activity in fabric data factory.

Now I would like to read that .csv file, format some of the columns and then append this into a warehouse table (I created this table already)

any suggestions please?

thanks

[v-cboorla-msft]

Hi @arkiboys 

Thanks for using Microsoft Fabric Community.

You can read the .csv file from azure storage container and append it to Fabric Warehouse by using Copy activity in Fabric Pipeline.

I hope this information helps. Please do let us know if you have any further queries.

Thank you.

[arkiboys]

I see...

In that case, question please:

1- in the destination tab - pre-script - before data is inserted into a warehouse table, get the filename (I know I can get the filename at source copy activity using $$FILENAME but not sure how to pass it into the pre-script box in destination tab.

thanks

[v-cboorla-msft · Accepted solution]

Hi @arkiboys 

I would to like to understand what are you trying to achieve with the help of pre-script? If you are looking to adding the additional column value to warehouse, then you simply continue with mapping option. 

If your query is different, please share more details of your query for better understanding.

Thank you.

[arkiboys]

 basically there is a table in the warehouse I created called tblwh The ADF pipeline reads the csv and places the data including the additional columnsi I added in th ecopy activity source nto this table tblwh. All good

Now if I re-run the pipeline I do not want the data to be duplicated into the tblwh. So I want to make sure it is deleted if it is already inserted in the previous pipeline run for the same filename.

and so I would like to have in the pre-scrip box to have something like:

delete from tblwh where filename = 'xyz'

[v-cboorla-msft]

Hi @arkiboys 

I was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

I can find you were able to get a resolution in this thread : Solved: copy activity - pre-copy script - Microsoft Fabric Community

Thanks.

[v-cboorla-msft]

Hi @arkiboys 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thanks.

[arkiboys]

all good now

using the presqlscript does the job

thank you