Forum Discussion
Working with time-based IOT data in PowerBi - advice needed
I'm new to PowerBI and struggling mightily to get it to show me 'simple' queries on my data. I'm used to writing LINQ queries in C# and probably suffering from trying to map this mindset onto DAX. I also suspect that the kind of 'native' support accumulation of state across time-based data just isn't there and I'm going to need to build a framework of custom functions to support what I want.
Hence, I'm looking for suggestions on the best approach to start querying my dataset.
The basic format of all my tables is
SensorId | Timestamp | attribute0 | attribute 1 | .... | attribute N
To give a concrete example of an 'events' table (you can assume the timestamp column just contains monotonically increasing
values)
SensorId | Timestamp | EventType sensorA | ... | Startup sensorB | ... | Startup sensorA | ... | Shutdown SensorB | ... | Alarm sensorA | ... | Startup SensorB | ... | Shutdown
And here's an associated (related) 'temperatures' Table
SensorId | Timestamp | Temperature sensorA | ... | 20
sensorB | ... | 21 sensorA | ... | 22
sensorB | ... | 19
The kinds of queries I'd like to ask are:
- Show me the latest eventType for each sensor (in the current time-slice)
- Show me the distribution of startup-to-shutdown times
- Show me the distribution of alarms by temperature.
One of the key concepts here is that each sensor is accumulating state over time- the temperature reports are at different times from the event reports, but we can assume that the current temperature of a sensor is whatever temperature it last reported. Almost all the queries I want to write rely on comparing a row against this 'accumulated state'.
I've worked out that creating a new calculated column is probably the best way of preparing the data for visualisation and I understand enough to write a DAX expression using the CALCULATE and FILTER functions that gives me (for example) the timestamp of the previous row for a particular sensor but it's getting really tedious endlessly repeating this pattern (not to mention the resulting DAX expression is pretty ugly and error-prone).
Conceptually what I'm looking for is a built-in or custom operation that would allow me to create these kinds of calculated columns....
- previousEvent = MostRecentBySensor(events[eventType])
- temperatureAtTimeOfEvent = MostRecentBySensor(temperatures[temperature])
- timeSinceStartup = MostRecentBySensorWithCondition(events[timestamp],events[eventType]="Startup")
In LINQ the implementation of MostRecentBySensorWithCondition is fairly easy...
var newColumnValue =
allRows
.Where(row=>row.SensorId== currentRow.SensorId
.Where(row=>row.timestamp < currentRow.timestamp)
.Where(_condition(row))
.OrderByDescending(row=>row.timestamp)
.Select(row=> _project(row))
.FirstOrDefault()
where _condition and _project are lambdas/functions that can be passed in to operate on the row being evaluated.
In DAX (or M?) I'm struggling to find the right answer. I think 'Where' generally translates to 'FILTER' but I can't see any DAX equivalent of just taking the the first/last row in an ordered result set so the DAX version probably needs to work out the most exact most recent timestamp (according the criteria) then use this to select the actual row we want (as a filter?) then use CALCULATE to simply take the (single or none) time in the result.
I can get this working for simple stuff but it seems completely unwieldy once we start to apply more interesting conditions. At this stage the whole experience is so painful I'm tempted just to go back and write code in the data-generator that would pre-populate columns for the queries I know I want to make.
Anway, if you've read this far... thank you! It's entirely possibly I'm heading into the weeds here and there is much easier way to accomplish what I want. I had expected that querying time-series data would be such a common task that there would be built-in operations. Perhaps there are but I've just overlooked them?
2 Replies
- neilmacmullenFrequent Visitor
Thanks Sam. The TOPN/MAXX combination looks like the 'missing link' to emulate the OrderBy/FirstOrDefault part of my query. My next problem is to make this reusable so I can parameterise the table/column names and filter condition. I suspect this means diving into M and finding equivalents of those operations. Will post here if I make progress.