Working with mass data

[GilesWalker]

Hi everyone,

Not sure where to post this so hoping you can help. I have been tasked with linking data from our locomotives to what is on the manifest. The data from our locomotives is contained in a database and has roughly twenty different columns and a row for every second. This means that in a 24 hour block there are 86,400 rows for each of the 17 locomotives. The data is captured in 24 hour blocks.

The database containing the manifest is seperate to the locomotive data, however their are two matching data points; the locomotive, number 1-17, and a date/time stamp, e.g. 20/05/2016 07:31 AM. So matching the two should be relatively straight forward in terms of relationships. My concern is the shear amount of data and trying to make sense of it. 

Are there any working examples out their where I can see how someone has set up a PBIX file which deals with mass data based off of time?

Or has anyone dealt with this that can give me some tips?

Thanks,

Giles

[Greg_Deckler]

So, I have worked with datasets in the millions of rows (3-5 million) without much issue in Power BI. However, if you want to analyze that much data, about 1.5 million rows per day, you may have to look at connecting to your data in "live" or Direct Query mode so that you are not actually importing the data into a dataset but rather using "live" querying. You may need to pull your different data sources into a cube and then connect live to the cube (SSAS).