Forum Discussion

alee023's avatar
alee023
Frequent Visitor
2 years ago
Solved

Same data transformations to be done on multiple files?

I receive a CSV file on a monthly basis, and I would like to perform the same exact transformation on every new file I receieve. For example, keep certain columns, do calculations based on those colu...
Dataflows
Need Help
  • v-cboorla-msft's avatar
    v-cboorla-msft
    2 years ago

    Hi alee023 

     

    Thanks for using Microsoft Fabric Community.

    As I understand that you want to use the same type of transformations on every new .CSV file that you will receive on monthly basis.

    If the schema of all the .CSV files are similar, then the two scenarios which you have mentioned above would work.

    For your scenario, if simplicity is your main priority Option 1 (Dataflow with SharePoint Source) might be sufficient. No matter which approach you choose, you'll achieve the desired result of automatically transforming your monthly CSV files while maintaining a clear separation between data source management (SharePoint) and data transformation logic (Dataflow).

     

    I hope this information helps. If you have any further questions please do let us know.

     

    Thanks.

     

     

v-cboorla-msft's avatar
v-cboorla-msft
Microsoft Employee
2 years ago

Hi alee023 

 

Thanks for using Microsoft Fabric Community.

As I understand that you want to use the same type of transformations on every new .CSV file that you will receive on monthly basis.

If the schema of all the .CSV files are similar, then the two scenarios which you have mentioned above would work.

For your scenario, if simplicity is your main priority Option 1 (Dataflow with SharePoint Source) might be sufficient. No matter which approach you choose, you'll achieve the desired result of automatically transforming your monthly CSV files while maintaining a clear separation between data source management (SharePoint) and data transformation logic (Dataflow).

 

I hope this information helps. If you have any further questions please do let us know.

 

Thanks.

 

 

divyamalhotra12's avatar
divyamalhotra12
Frequent Visitor
2 years ago

Hi,

I am facing a similar issue. I have a dataset wherein I applied a few transformations such as change data type for all columns, choose columns etc. Now, I want the data to be updated and the same transformations to be applied there. For instance, I have a file called, 'Orders'. Now, I have an updated version of the same file in sharepoint. How do I proceed with updating the same in dataflow and then ensuring the transformations get applied to it.

I wanted to understand the steps to perform the same. Could you please help me with the same.

  • alee023's avatar
    alee023
    Frequent Visitor
    2 years ago

    Here's what I did:

    1. My files were uploaded to SharePoint, so my dataflow source was to the sharepoint folder. I'm not sure how this works if it were OneDrive.
    2. Filter the dataset to the correct folder path - because the source doesn't allow you to connect to specific subfolders, so you deal with it in this step
    3. Filter the dataset to get the most recent file (sort by date created, descending)
    4. Find the 'Combine files' option in the menu. This will extract all the data from the file