Forum Discussion

neilmacmullen's avatar
neilmacmullen
Frequent Visitor
8 years ago

Working with time-based IOT data in PowerBi - advice needed


I'm new to PowerBI and struggling mightily to get it to show me 'simple' queries on my data.  I'm used to writing LINQ queries in C# and probably suffering from trying to map this mindset onto DAX.  I also suspect that the kind of 'native' support accumulation of state across  time-based data just isn't there and I'm going to need to build a framework of custom functions to support what I want.

 

Hence, I'm looking for suggestions on the best approach to start querying my dataset.

 

The basic format of all my tables is

SensorId | Timestamp | attribute0 | attribute 1 | .... | attribute N

To give a concrete example of an 'events' table (you can assume the timestamp column just contains monotonically increasing
values)

SensorId | Timestamp | EventType
sensorA | ... | Startup
sensorB | ... | Startup
sensorA | ... | Shutdown
SensorB | ... | Alarm
sensorA | ... | Startup
SensorB | ... | Shutdown

And here's an associated (related) 'temperatures' Table

SensorId | Timestamp | Temperature
sensorA | ... | 20
sensorB | ... | 21 sensorA | ... | 22
sensorB | ... | 19


The kinds of queries I'd like to ask are:

  • Show me the latest eventType for each sensor (in the current time-slice)
  • Show me the distribution of startup-to-shutdown times
  • Show me the distribution of alarms by temperature.

One of the key concepts here is that each sensor is accumulating state over time- the temperature reports are at different times from the event reports, but we can assume that the current temperature of a sensor is whatever temperature it last reported.  Almost all the queries I want to write rely on comparing a row against this 'accumulated state'.

 

I've worked out that creating a new calculated column is probably the best way of preparing the data for visualisation and I understand enough to write a DAX expression using the CALCULATE and FILTER functions that gives me (for example) the timestamp of the previous row for a particular sensor but it's getting really tedious endlessly repeating this pattern (not to mention the resulting DAX expression is pretty ugly and error-prone). 

 

Conceptually what I'm looking for is a built-in or custom operation that would allow me to create these kinds of calculated columns.... 

  • previousEvent = MostRecentBySensor(events[eventType])
  • temperatureAtTimeOfEvent = MostRecentBySensor(temperatures[temperature])
  • timeSinceStartup = MostRecentBySensorWithCondition(events[timestamp],events[eventType]="Startup")

In LINQ the implementation of MostRecentBySensorWithCondition is fairly easy...

var newColumnValue =       
    allRows  
    .Where(row=>row.SensorId== currentRow.SensorId    
    .Where(row=>row.timestamp < currentRow.timestamp)      
    .Where(_condition(row))   
.OrderByDescending(row=>row.timestamp) .Select(row=> _project(row)) .FirstOrDefault()


 
where _condition and _project are lambdas/functions that can be passed in to operate on the row being evaluated.

 

In DAX (or M?) I'm struggling to find the right answer. I think 'Where' generally translates to 'FILTER' but I can't see any DAX equivalent of just taking the the first/last row in an ordered result set so the DAX version probably needs to work out the most exact most recent timestamp (according the criteria) then use this to select the actual row we want (as a filter?) then use CALCULATE to simply take the (single or none) time in the result.

 

I can get this working for simple stuff but it seems completely unwieldy once we start to apply more interesting conditions.  At this stage the whole experience is so painful I'm tempted just to go back and write code in the data-generator that would pre-populate columns for the queries I know I want to make.

 

Anway, if you've read this far... thank you! It's entirely possibly I'm heading into the weeds here and there is much easier way to accomplish what I want.  I had expected that querying time-series data would be such a common task that there would be built-in operations.  Perhaps there are but I've just overlooked them?

 

2 Replies