Forum Discussion

toddysm's avatar
toddysm
New Member
9 years ago
Solved

Filtering and dynamically updating list of source CSV files

Hello,   We have jobs that runs every hour and generate CSV files that follow pattern similar to:   YYYYMMDD-HH_some-static-text.CSV Where some-static-text is specific to the job that is gener...
Need Help
  • v-sihou-msft's avatar
    v-sihou-msft
    9 years ago

    toddysm

     

    Since all your CSV files are under same folder, you can get data from folder.

     

     

    You will get the metadata of the folder. Add a text filter on Name column to keep expected files only.

     

     

     

     

    "Invoke Custom Function" the transform file from the folder.

     

     

    Now you just need to remove other columns and expand the "Table" column to get the combined dataset.

     

     

    Regards,

     

     

v-sihou-msft's avatar
v-sihou-msft
Microsoft Employee
9 years ago

toddysm

 

Since all your CSV files are under same folder, you can get data from folder.

 

 

You will get the metadata of the folder. Add a text filter on Name column to keep expected files only.

 

 

 

 

"Invoke Custom Function" the transform file from the folder.

 

 

Now you just need to remove other columns and expand the "Table" column to get the combined dataset.

 

 

Regards,

 

 

  • toddysm's avatar
    toddysm
    New Member
    9 years ago

    Thanks Simon-Hou,

     

    The confusion was that after filtering I wasn't seeing all the files I expected to see. Refreshing from the source solved this. Appreciate the suggestion for Custom Function though - this will come handy