Get Data from Folder of files with different column names

[anthrakia]

My data source is a folder of .csv files: data logs from a fleet of cars.  The configuration (i.e. column names) of the data logs can change from time to time.  The changes may be applied to some or all cars.  The folder gets refreshed manually from time to time as it is populated with new data logs.

I have visualizations set up with variables that exist on some configurations, but not on others.  That's OK for me - as I just expect the visualization to show blank for the data logs without the variable of interest.

The problem is that when I refresh my data, Power BI seems to be selecting the first file (alphanumeric order of the filename?) as a sample.  So, if it selects a sample which is missing a variable that is used in one my visualizations, my report "breaks".  And, the variable of interest is not even searchable as a field, even though I know that it exists on at least some of the files in the folder.

My questions:

1. Is the above-described behavior for getting data from a folder in Power BI expected?
2. If so, how can I work around this?  (I have thought about making a template file, naming it "0.csv", and ensuring that all possible column names across all of my data log configurations are captured in that file.  I'm looking for a more robust/elegant solution.)

[v-yulgu-msft]

Hi @anthrakia,

What does the "variables" mean as you mentioned? 

Each time, you get data from a folder containing multiple .csv files, how do you choose the wanted file? Could you please provide more description?

Regards,

Yuliana Gu

[anthrakia]

@v-yulgu-msft 

Thank you for your response.  My .csv files are pulled from a data logger installed on each car in our fleet.  The variables are organized in columns.  The first 3 columns are typically time variables: time stamp, time in milliseconds relative to the event trigger, time in milliseconds starting at zero.  Subsequent columns are other variables recorded by the data logger.  For example wheel speed, steering wheel angle, brake force, vehicle speed, etc.

Say, for example, I set up a visualization for wheel speed vs time.  Depending on my data logger configuration, some cars may not record wheel speed.  I would like that visualization to simply show blank.

The first time I load data into PowerBI from a folder, I get to choose the sample file.  i.e. First File or I can choose from a drop-down list.  But if I replace the folder of files with a new set of files and, or point the report to a different folder as a data source, PowerBI may choose a sample file that doesn't have wheel speed in it.  At that point, my report breaks, even if there are other files in the folder that include wheel speed.