Transform Data on .CSV Fiule Updates

[ghutchins]

I exported a .CSV file from a third party survey app, and shaped the data with 'Transform Data' and was able to produce a viz from that data.  Today, becuase there are new surverys...   I exported the .CSV file again and overwrote the source file for yesterday's viz.

Because the file is structurally the same, only new rows, that the data transformations i made on the first file would be applied to today's version of the file.  It did not happen, many errors indicating missing columns.  

What did I do wrong?  Do I need to do the same transformations on each file version?  Any help is apprecitated.

[v-alq-msft]

Hi, @ghutchins 

If you take the answer of someone, please mark it as the solution to help the other members who have same problems find it more quickly. If not, let me know and I'll try to help you further. Thanks.

Best Regards

Allan

[v-alq-msft]

Hi, @ghutchins 

Based on your description, I created data to reproduce your scenario.

1.csv:

2.csv:

You may use Text/CSV connector to connect to 1.csv and filter out rows where 'Age' is greater than or equal to 12. Then you can create a parameter named as 'Filepath', go to 'Advanced Editor', replace the path with the parameter.

Then you may right click the first query and click 'create function' to create a function named 'TransformCSV'.

Finally you can input the filepath to apply the same transfromation the .csv files.

Best Regards

Allan

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[camargos88]

Hi @ghutchins ,

You should use the folder connection. So you can use many files and apply transformation to them.

You have to use the Transformation Sample File to apply the transformation, so it will apply to each file.

Just be aware, they have to have the same data structure.