topic copy .csv data to warehouse in Pipelines

copy .csv data to warehouse

Anonymous — Fri, 10 May 2024 08:51:21 GMT

hello,

using the copy activity in fabric I can take data from .csv file (Which is sitting in azure storage account) and place the data into the table in the warehouse (tblwh) - I am using mapping in the copy activity. all good so far.

Now I have added a few more columns into the tblwh. These new columns are not in the source .csv file.

For example, I have added a new column called presentdate, filepath, fileextension, fullfilepath

Question:

How is it possible I populate these extra columns in the tblwh because the mapping tab in copy activity does not obviously have any source columns for these ?

thanks

Re: copy .csv data to warehouse

Anonymous — Fri, 10 May 2024 09:19:30 GMT

Hi Anonymous ,

Thanks for using Fabric Community.
You can add your additional columns in Copy Activity from here -

Eg:

After adding additional columns in source can you please click on import schema in mapping?

Eg:

After importing you can run the pipeline

Hope this helps.

Re: copy .csv data to warehouse

Anonymous — Fri, 10 May 2024 09:27:03 GMT

this is good. thanks

in source copy tab, I use

filename --> $$FILENAME which gives 'some-03-May-2024.csv' --> this is correct

directory_path --> $$FILEPATH but this gives the fullpath such as 'Data/some-new/some-03-May-2024.csv'

whereas I would like to see the directory such as 'Data/some-new'

thanks

Re: copy .csv data to warehouse

Anonymous — Fri, 10 May 2024 09:43:50 GMT

Hi Anonymous ,

I can also find the same result -

By default FileName and FilePath are designed like that.
As a workaround (assume file path won't change dynamically), you can use custom value -

Hope this is helpful. Please let me know incase of further queries.

Re: copy .csv data to warehouse

Anonymous — Fri, 10 May 2024 09:48:01 GMT

hi, I see you have hardcoded the value however my pipeline I would like it to be dynamic as the path will vary since the source varies

any suggestions?

thanks

Re: copy .csv data to warehouse

Anonymous — Fri, 10 May 2024 10:09:41 GMT

Hi Anonymous ,

Can you please share the screenshot of Copy Activity - Source?
I would like to see how are defining file path dynamically? So that I can help you.

Re: copy .csv data to warehouse

Anonymous — Fri, 10 May 2024 10:11:39 GMT

thank you

Re: copy .csv data to warehouse

Anonymous — Fri, 10 May 2024 10:13:33 GMT

Hi Anonymous ,

Glad to know that your query got resolved. Please continue using Fabric Community on your further queries.