Forum Discussion
Power Bi Challenge - Round 2
Hello Community Champions!
This is definitely not a common requirement but it is a real one. People working in the industrial sectors would probably encounter similar situations. Engineering companies who design Central Monitoring Systems or Building Management Systems usually design the shape of the output data in a way that makes it easy to handle with Excel. However, today more and more businesses are shifting to Power Bi. Industrial firms are no exception specially that they’re having probably more concerns about their data security.
Enough of introduction.. the challenge this time is about handling aggregations in horizontally expanded tables. Tables that contain huge number of value columns. In our case the CMS is recording a temperature measurement every 10 minutes accumulating 144 columns in a single row every day.
Power Query experts do not get excited.. The table, which is supposed to be retrieved via a Live Connection due to data control measures, contains a row for each temperature sensor each day for the current year 2024. Everyday one more row is added to each sensor. The 144 columns contains the measurement of the temperature recorded every 10 minutes. This daily variation in the temperature is important, however this is not our problem for today’s challenge.
Today we are interested in Daily Averages. The requirement is simple: Create a line chart of the average daily temperature against [Date] in the x-axis.
Attached is the sample data file.
@lbendlin @Greg_Deckler @Jihwan_Kim @sjoerdvn @OwenAuger @DataNinja777 @quantumudit @ThxAlot @johnbasha33 @Dangar332 @ValtteriN @AntrikshSharma @AlexisOlson @Martin_D
tamerj1 Hey, here's a version that doesn't require hardcoding and you can theoretically add as many pre-columns as you want. Basically Measure 2 but with a dynamic finding of the number of commas before the first value column.
Measure 4 = VAR __Text = SUBSTITUTE( TOCSV( 'T_MEASURES', , ",", 1 ), UNICHAR(10), "|" ) VAR __1 = PATHITEM( __Text, 1 ) VAR __FirstColon = FIND(":", __1) VAR __Left = LEFT( __1, __FirstColon ) VAR __Commas = LEN( __Left ) - LEN( SUBSTITUTE( __Left, ",", "" ) ) VAR __2 = PATHITEM( __Text, 2 ) VAR __3 = SUBSTITUTE( __2, ",", "|", __Commas ) VAR __4 = PATHITEM( __3, 2 ) VAR __Path = SUBSTITUTE( __4, ",", "|" ) VAR __Table = ADDCOLUMNS( GENERATESERIES(1, 144, 1), "__Value", PATHITEM( __Path, [Value]) + 0 ) VAR __Result = AVERAGEX( __Table, [__Value] ) RETURN __Resulttamerj1 OK, one "final" version. This one *should* work with multiple rows of sensor data on the same day and doesn't assume basically anything in terms of how many columns of data, just that the first column of data that the name includes a ":" and that after that the data columns are sequential with no non-data columns mixed in.
I don't have the original data file so I can't actually test it with multiple data on the same day (although that doesn't make sense in this scenario it might make sense in other similar circumstances or scenarios so I wanted to try doing it).
Measure 5 = // Deal with the header to figure out what columns need to be removed VAR __TableWHeaders = SUBSTITUTE( TOCSV( 'T_MEASURES', , ",", 1 ), UNICHAR(10), "|" ) // TOCSV with Headers VAR __Headers = PATHITEM( __TableWHeaders, 1 ) //headers VAR __FirstColon = FIND(":", __Headers) // find the first colon, which should be the first column of data VAR __Left = LEFT( __Headers, __FirstColon ) // grab everything to the left of the first colon VAR __Commas = LEN( __Left ) - LEN( SUBSTITUTE( __Left, ",", "" ) ) // figure out how many commas (columns) are in the "pre-columns" before the data // Now deal with the data which might have multiple rows VAR __TableWOHeaders = SUBSTITUTE( TOCSV( 'T_MEASURES', , ",", 0 ), UNICHAR(10), "|" ) // TOCSV without Headers VAR __Count = COUNTROWS('T_MEASURES') // number of rows in table VAR __Data = // Generate a table with a column that strips out the pre-columns ADDCOLUMNS( GENERATESERIES(1, __Count, 1), "__Data", VAR __Text = PATHITEM( __TableWOHeaders, [Value]) VAR __Path = SUBSTITUTE( __Text, ",", "|", __Commas ) VAR __Result = PATHITEM( __Path, 2 ) RETURN __Result ) VAR __DataText = CONCATENATEX( __Data, [__Data], "|" ) // convert the table back into a text string with each row as a path item VAR __Path = SUBSTITUTE( __DataText, ",", "|" ) // convert all columns to path items VAR __DataColumns = LEN( __DataText) - LEN( SUBSTITUTE( __DataText, ",", "" ) ) // figure out how many columns of data we are dealing with VAR __Table = // Convert the path into a table ADDCOLUMNS( GENERATESERIES(1, __DataColumns, 1), "__Value", PATHITEM( __Path, [Value]) + 0 ) VAR __Result = AVERAGEX( __Table, [__Value] ) // run an aggregation against the table RETURN __ResultThe other obvious hard-coded solution (manual unpivoting -- shown below) works too. It has more overhead but performance is still acceptable. The overhead is largely fixed and performance is barely affected by changing from one month of data to five months of data, unlike the text-parsing solutions whose timing scales roughly linearly with the number of months of data selected and quickly become impractical for larger amounts of data.
Testing twenty cold cache trials on the full set of data (still only 876 rows):
My simple solution above:A more general unpivot solution (below):
tamerj1's solution from above:
Avg Temp (Unpivot) = VAR __Unpivot__ = UNION ( SELECTCOLUMNS ( T, "@Time", "00:00", "@Temp", T[00:00] ), SELECTCOLUMNS ( T, "@Time", "00:10", "@Temp", T[00:10] ), SELECTCOLUMNS ( T, "@Time", "00:20", "@Temp", T[00:20] ), SELECTCOLUMNS ( T, "@Time", "00:30", "@Temp", T[00:30] ), SELECTCOLUMNS ( T, "@Time", "00:40", "@Temp", T[00:40] ), SELECTCOLUMNS ( T, "@Time", "00:50", "@Temp", T[00:50] ), SELECTCOLUMNS ( T, "@Time", "01:00", "@Temp", T[01:00] ), SELECTCOLUMNS ( T, "@Time", "01:10", "@Temp", T[01:10] ), SELECTCOLUMNS ( T, "@Time", "01:20", "@Temp", T[01:20] ), SELECTCOLUMNS ( T, "@Time", "01:30", "@Temp", T[01:30] ), SELECTCOLUMNS ( T, "@Time", "01:40", "@Temp", T[01:40] ), SELECTCOLUMNS ( T, "@Time", "01:50", "@Temp", T[01:50] ), SELECTCOLUMNS ( T, "@Time", "02:00", "@Temp", T[02:00] ), SELECTCOLUMNS ( T, "@Time", "02:10", "@Temp", T[02:10] ), SELECTCOLUMNS ( T, "@Time", "02:20", "@Temp", T[02:20] ), SELECTCOLUMNS ( T, "@Time", "02:30", "@Temp", T[02:30] ), SELECTCOLUMNS ( T, "@Time", "02:40", "@Temp", T[02:40] ), SELECTCOLUMNS ( T, "@Time", "02:50", "@Temp", T[02:50] ), SELECTCOLUMNS ( T, "@Time", "03:00", "@Temp", T[03:00] ), SELECTCOLUMNS ( T, "@Time", "03:10", "@Temp", T[03:10] ), SELECTCOLUMNS ( T, "@Time", "03:20", "@Temp", T[03:20] ), SELECTCOLUMNS ( T, "@Time", "03:30", "@Temp", T[03:30] ), SELECTCOLUMNS ( T, "@Time", "03:40", "@Temp", T[03:40] ), SELECTCOLUMNS ( T, "@Time", "03:50", "@Temp", T[03:50] ), SELECTCOLUMNS ( T, "@Time", "04:00", "@Temp", T[04:00] ), SELECTCOLUMNS ( T, "@Time", "04:10", "@Temp", T[04:10] ), SELECTCOLUMNS ( T, "@Time", "04:20", "@Temp", T[04:20] ), SELECTCOLUMNS ( T, "@Time", "04:30", "@Temp", T[04:30] ), SELECTCOLUMNS ( T, "@Time", "04:40", "@Temp", T[04:40] ), SELECTCOLUMNS ( T, "@Time", "04:50", "@Temp", T[04:50] ), SELECTCOLUMNS ( T, "@Time", "05:00", "@Temp", T[05:00] ), SELECTCOLUMNS ( T, "@Time", "05:10", "@Temp", T[05:10] ), SELECTCOLUMNS ( T, "@Time", "05:20", "@Temp", T[05:20] ), SELECTCOLUMNS ( T, "@Time", "05:30", "@Temp", T[05:30] ), SELECTCOLUMNS ( T, "@Time", "05:40", "@Temp", T[05:40] ), SELECTCOLUMNS ( T, "@Time", "05:50", "@Temp", T[05:50] ), SELECTCOLUMNS ( T, "@Time", "06:00", "@Temp", T[06:00] ), SELECTCOLUMNS ( T, "@Time", "06:10", "@Temp", T[06:10] ), SELECTCOLUMNS ( T, "@Time", "06:20", "@Temp", T[06:20] ), SELECTCOLUMNS ( T, "@Time", "06:30", "@Temp", T[06:30] ), SELECTCOLUMNS ( T, "@Time", "06:40", "@Temp", T[06:40] ), SELECTCOLUMNS ( T, "@Time", "06:50", "@Temp", T[06:50] ), SELECTCOLUMNS ( T, "@Time", "07:00", "@Temp", T[07:00] ), SELECTCOLUMNS ( T, "@Time", "07:10", "@Temp", T[07:10] ), SELECTCOLUMNS ( T, "@Time", "07:20", "@Temp", T[07:20] ), SELECTCOLUMNS ( T, "@Time", "07:30", "@Temp", T[07:30] ), SELECTCOLUMNS ( T, "@Time", "07:40", "@Temp", T[07:40] ), SELECTCOLUMNS ( T, "@Time", "07:50", "@Temp", T[07:50] ), SELECTCOLUMNS ( T, "@Time", "08:00", "@Temp", T[08:00] ), SELECTCOLUMNS ( T, "@Time", "08:10", "@Temp", T[08:10] ), SELECTCOLUMNS ( T, "@Time", "08:20", "@Temp", T[08:20] ), SELECTCOLUMNS ( T, "@Time", "08:30", "@Temp", T[08:30] ), SELECTCOLUMNS ( T, "@Time", "08:40", "@Temp", T[08:40] ), SELECTCOLUMNS ( T, "@Time", "08:50", "@Temp", T[08:50] ), SELECTCOLUMNS ( T, "@Time", "09:00", "@Temp", T[09:00] ), SELECTCOLUMNS ( T, "@Time", "09:10", "@Temp", T[09:10] ), SELECTCOLUMNS ( T, "@Time", "09:20", "@Temp", T[09:20] ), SELECTCOLUMNS ( T, "@Time", "09:30", "@Temp", T[09:30] ), SELECTCOLUMNS ( T, "@Time", "09:40", "@Temp", T[09:40] ), SELECTCOLUMNS ( T, "@Time", "09:50", "@Temp", T[09:50] ), SELECTCOLUMNS ( T, "@Time", "10:00", "@Temp", T[10:00] ), SELECTCOLUMNS ( T, "@Time", "10:10", "@Temp", T[10:10] ), SELECTCOLUMNS ( T, "@Time", "10:20", "@Temp", T[10:20] ), SELECTCOLUMNS ( T, "@Time", "10:30", "@Temp", T[10:30] ), SELECTCOLUMNS ( T, "@Time", "10:40", "@Temp", T[10:40] ), SELECTCOLUMNS ( T, "@Time", "10:50", "@Temp", T[10:50] ), SELECTCOLUMNS ( T, "@Time", "11:00", "@Temp", T[11:00] ), SELECTCOLUMNS ( T, "@Time", "11:10", "@Temp", T[11:10] ), SELECTCOLUMNS ( T, "@Time", "11:20", "@Temp", T[11:20] ), SELECTCOLUMNS ( T, "@Time", "11:30", "@Temp", T[11:30] ), SELECTCOLUMNS ( T, "@Time", "11:40", "@Temp", T[11:40] ), SELECTCOLUMNS ( T, "@Time", "11:50", "@Temp", T[11:50] ), SELECTCOLUMNS ( T, "@Time", "12:00", "@Temp", T[12:00] ), SELECTCOLUMNS ( T, "@Time", "12:10", "@Temp", T[12:10] ), SELECTCOLUMNS ( T, "@Time", "12:20", "@Temp", T[12:20] ), SELECTCOLUMNS ( T, "@Time", "12:30", "@Temp", T[12:30] ), SELECTCOLUMNS ( T, "@Time", "12:40", "@Temp", T[12:40] ), SELECTCOLUMNS ( T, "@Time", "12:50", "@Temp", T[12:50] ), SELECTCOLUMNS ( T, "@Time", "13:00", "@Temp", T[13:00] ), SELECTCOLUMNS ( T, "@Time", "13:10", "@Temp", T[13:10] ), SELECTCOLUMNS ( T, "@Time", "13:20", "@Temp", T[13:20] ), SELECTCOLUMNS ( T, "@Time", "13:30", "@Temp", T[13:30] ), SELECTCOLUMNS ( T, "@Time", "13:40", "@Temp", T[13:40] ), SELECTCOLUMNS ( T, "@Time", "13:50", "@Temp", T[13:50] ), SELECTCOLUMNS ( T, "@Time", "14:00", "@Temp", T[14:00] ), SELECTCOLUMNS ( T, "@Time", "14:10", "@Temp", T[14:10] ), SELECTCOLUMNS ( T, "@Time", "14:20", "@Temp", T[14:20] ), SELECTCOLUMNS ( T, "@Time", "14:30", "@Temp", T[14:30] ), SELECTCOLUMNS ( T, "@Time", "14:40", "@Temp", T[14:40] ), SELECTCOLUMNS ( T, "@Time", "14:50", "@Temp", T[14:50] ), SELECTCOLUMNS ( T, "@Time", "15:00", "@Temp", T[15:00] ), SELECTCOLUMNS ( T, "@Time", "15:10", "@Temp", T[15:10] ), SELECTCOLUMNS ( T, "@Time", "15:20", "@Temp", T[15:20] ), SELECTCOLUMNS ( T, "@Time", "15:30", "@Temp", T[15:30] ), SELECTCOLUMNS ( T, "@Time", "15:40", "@Temp", T[15:40] ), SELECTCOLUMNS ( T, "@Time", "15:50", "@Temp", T[15:50] ), SELECTCOLUMNS ( T, "@Time", "16:00", "@Temp", T[16:00] ), SELECTCOLUMNS ( T, "@Time", "16:10", "@Temp", T[16:10] ), SELECTCOLUMNS ( T, "@Time", "16:20", "@Temp", T[16:20] ), SELECTCOLUMNS ( T, "@Time", "16:30", "@Temp", T[16:30] ), SELECTCOLUMNS ( T, "@Time", "16:40", "@Temp", T[16:40] ), SELECTCOLUMNS ( T, "@Time", "16:50", "@Temp", T[16:50] ), SELECTCOLUMNS ( T, "@Time", "17:00", "@Temp", T[17:00] ), SELECTCOLUMNS ( T, "@Time", "17:10", "@Temp", T[17:10] ), SELECTCOLUMNS ( T, "@Time", "17:20", "@Temp", T[17:20] ), SELECTCOLUMNS ( T, "@Time", "17:30", "@Temp", T[17:30] ), SELECTCOLUMNS ( T, "@Time", "17:40", "@Temp", T[17:40] ), SELECTCOLUMNS ( T, "@Time", "17:50", "@Temp", T[17:50] ), SELECTCOLUMNS ( T, "@Time", "18:00", "@Temp", T[18:00] ), SELECTCOLUMNS ( T, "@Time", "18:10", "@Temp", T[18:10] ), SELECTCOLUMNS ( T, "@Time", "18:20", "@Temp", T[18:20] ), SELECTCOLUMNS ( T, "@Time", "18:30", "@Temp", T[18:30] ), SELECTCOLUMNS ( T, "@Time", "18:40", "@Temp", T[18:40] ), SELECTCOLUMNS ( T, "@Time", "18:50", "@Temp", T[18:50] ), SELECTCOLUMNS ( T, "@Time", "19:00", "@Temp", T[19:00] ), SELECTCOLUMNS ( T, "@Time", "19:10", "@Temp", T[19:10] ), SELECTCOLUMNS ( T, "@Time", "19:20", "@Temp", T[19:20] ), SELECTCOLUMNS ( T, "@Time", "19:30", "@Temp", T[19:30] ), SELECTCOLUMNS ( T, "@Time", "19:40", "@Temp", T[19:40] ), SELECTCOLUMNS ( T, "@Time", "19:50", "@Temp", T[19:50] ), SELECTCOLUMNS ( T, "@Time", "20:00", "@Temp", T[20:00] ), SELECTCOLUMNS ( T, "@Time", "20:10", "@Temp", T[20:10] ), SELECTCOLUMNS ( T, "@Time", "20:20", "@Temp", T[20:20] ), SELECTCOLUMNS ( T, "@Time", "20:30", "@Temp", T[20:30] ), SELECTCOLUMNS ( T, "@Time", "20:40", "@Temp", T[20:40] ), SELECTCOLUMNS ( T, "@Time", "20:50", "@Temp", T[20:50] ), SELECTCOLUMNS ( T, "@Time", "21:00", "@Temp", T[21:00] ), SELECTCOLUMNS ( T, "@Time", "21:10", "@Temp", T[21:10] ), SELECTCOLUMNS ( T, "@Time", "21:20", "@Temp", T[21:20] ), SELECTCOLUMNS ( T, "@Time", "21:30", "@Temp", T[21:30] ), SELECTCOLUMNS ( T, "@Time", "21:40", "@Temp", T[21:40] ), SELECTCOLUMNS ( T, "@Time", "21:50", "@Temp", T[21:50] ), SELECTCOLUMNS ( T, "@Time", "22:00", "@Temp", T[22:00] ), SELECTCOLUMNS ( T, "@Time", "22:10", "@Temp", T[22:10] ), SELECTCOLUMNS ( T, "@Time", "22:20", "@Temp", T[22:20] ), SELECTCOLUMNS ( T, "@Time", "22:30", "@Temp", T[22:30] ), SELECTCOLUMNS ( T, "@Time", "22:40", "@Temp", T[22:40] ), SELECTCOLUMNS ( T, "@Time", "22:50", "@Temp", T[22:50] ), SELECTCOLUMNS ( T, "@Time", "23:00", "@Temp", T[23:00] ), SELECTCOLUMNS ( T, "@Time", "23:10", "@Temp", T[23:10] ), SELECTCOLUMNS ( T, "@Time", "23:20", "@Temp", T[23:20] ), SELECTCOLUMNS ( T, "@Time", "23:30", "@Temp", T[23:30] ), SELECTCOLUMNS ( T, "@Time", "23:40", "@Temp", T[23:40] ), SELECTCOLUMNS ( T, "@Time", "23:50", "@Temp", T[23:50] ) ) VAR __Result = AVERAGEX ( __Unpivot__, [@Temp] ) RETURN __ResultAnonymous I thought about an alternate solution instead of DAX.
@tamerj1 I have though about an alternate approach to solve the above challenge.
As in the original fole we do not have the option to model the data then we publish it and we set the maximum amount of refresh's possible(depending on licence)
We open a new .pbix file and then through XMLA endpoint we bring the published data in the new .pbix file. Where we unpivot all the columns with the values so we get rid of the multi column aggregations. From that on it pretty simple to build the visuals.
I have cheated a bit as it has not been solved with DAX but it works.I am also attaching a video link of the steps how to do it.
26 Replies
The other obvious hard-coded solution (manual unpivoting -- shown below) works too. It has more overhead but performance is still acceptable. The overhead is largely fixed and performance is barely affected by changing from one month of data to five months of data, unlike the text-parsing solutions whose timing scales roughly linearly with the number of months of data selected and quickly become impractical for larger amounts of data.
Testing twenty cold cache trials on the full set of data (still only 876 rows):
My simple solution above:A more general unpivot solution (below):
tamerj1's solution from above:
Avg Temp (Unpivot) = VAR __Unpivot__ = UNION ( SELECTCOLUMNS ( T, "@Time", "00:00", "@Temp", T[00:00] ), SELECTCOLUMNS ( T, "@Time", "00:10", "@Temp", T[00:10] ), SELECTCOLUMNS ( T, "@Time", "00:20", "@Temp", T[00:20] ), SELECTCOLUMNS ( T, "@Time", "00:30", "@Temp", T[00:30] ), SELECTCOLUMNS ( T, "@Time", "00:40", "@Temp", T[00:40] ), SELECTCOLUMNS ( T, "@Time", "00:50", "@Temp", T[00:50] ), SELECTCOLUMNS ( T, "@Time", "01:00", "@Temp", T[01:00] ), SELECTCOLUMNS ( T, "@Time", "01:10", "@Temp", T[01:10] ), SELECTCOLUMNS ( T, "@Time", "01:20", "@Temp", T[01:20] ), SELECTCOLUMNS ( T, "@Time", "01:30", "@Temp", T[01:30] ), SELECTCOLUMNS ( T, "@Time", "01:40", "@Temp", T[01:40] ), SELECTCOLUMNS ( T, "@Time", "01:50", "@Temp", T[01:50] ), SELECTCOLUMNS ( T, "@Time", "02:00", "@Temp", T[02:00] ), SELECTCOLUMNS ( T, "@Time", "02:10", "@Temp", T[02:10] ), SELECTCOLUMNS ( T, "@Time", "02:20", "@Temp", T[02:20] ), SELECTCOLUMNS ( T, "@Time", "02:30", "@Temp", T[02:30] ), SELECTCOLUMNS ( T, "@Time", "02:40", "@Temp", T[02:40] ), SELECTCOLUMNS ( T, "@Time", "02:50", "@Temp", T[02:50] ), SELECTCOLUMNS ( T, "@Time", "03:00", "@Temp", T[03:00] ), SELECTCOLUMNS ( T, "@Time", "03:10", "@Temp", T[03:10] ), SELECTCOLUMNS ( T, "@Time", "03:20", "@Temp", T[03:20] ), SELECTCOLUMNS ( T, "@Time", "03:30", "@Temp", T[03:30] ), SELECTCOLUMNS ( T, "@Time", "03:40", "@Temp", T[03:40] ), SELECTCOLUMNS ( T, "@Time", "03:50", "@Temp", T[03:50] ), SELECTCOLUMNS ( T, "@Time", "04:00", "@Temp", T[04:00] ), SELECTCOLUMNS ( T, "@Time", "04:10", "@Temp", T[04:10] ), SELECTCOLUMNS ( T, "@Time", "04:20", "@Temp", T[04:20] ), SELECTCOLUMNS ( T, "@Time", "04:30", "@Temp", T[04:30] ), SELECTCOLUMNS ( T, "@Time", "04:40", "@Temp", T[04:40] ), SELECTCOLUMNS ( T, "@Time", "04:50", "@Temp", T[04:50] ), SELECTCOLUMNS ( T, "@Time", "05:00", "@Temp", T[05:00] ), SELECTCOLUMNS ( T, "@Time", "05:10", "@Temp", T[05:10] ), SELECTCOLUMNS ( T, "@Time", "05:20", "@Temp", T[05:20] ), SELECTCOLUMNS ( T, "@Time", "05:30", "@Temp", T[05:30] ), SELECTCOLUMNS ( T, "@Time", "05:40", "@Temp", T[05:40] ), SELECTCOLUMNS ( T, "@Time", "05:50", "@Temp", T[05:50] ), SELECTCOLUMNS ( T, "@Time", "06:00", "@Temp", T[06:00] ), SELECTCOLUMNS ( T, "@Time", "06:10", "@Temp", T[06:10] ), SELECTCOLUMNS ( T, "@Time", "06:20", "@Temp", T[06:20] ), SELECTCOLUMNS ( T, "@Time", "06:30", "@Temp", T[06:30] ), SELECTCOLUMNS ( T, "@Time", "06:40", "@Temp", T[06:40] ), SELECTCOLUMNS ( T, "@Time", "06:50", "@Temp", T[06:50] ), SELECTCOLUMNS ( T, "@Time", "07:00", "@Temp", T[07:00] ), SELECTCOLUMNS ( T, "@Time", "07:10", "@Temp", T[07:10] ), SELECTCOLUMNS ( T, "@Time", "07:20", "@Temp", T[07:20] ), SELECTCOLUMNS ( T, "@Time", "07:30", "@Temp", T[07:30] ), SELECTCOLUMNS ( T, "@Time", "07:40", "@Temp", T[07:40] ), SELECTCOLUMNS ( T, "@Time", "07:50", "@Temp", T[07:50] ), SELECTCOLUMNS ( T, "@Time", "08:00", "@Temp", T[08:00] ), SELECTCOLUMNS ( T, "@Time", "08:10", "@Temp", T[08:10] ), SELECTCOLUMNS ( T, "@Time", "08:20", "@Temp", T[08:20] ), SELECTCOLUMNS ( T, "@Time", "08:30", "@Temp", T[08:30] ), SELECTCOLUMNS ( T, "@Time", "08:40", "@Temp", T[08:40] ), SELECTCOLUMNS ( T, "@Time", "08:50", "@Temp", T[08:50] ), SELECTCOLUMNS ( T, "@Time", "09:00", "@Temp", T[09:00] ), SELECTCOLUMNS ( T, "@Time", "09:10", "@Temp", T[09:10] ), SELECTCOLUMNS ( T, "@Time", "09:20", "@Temp", T[09:20] ), SELECTCOLUMNS ( T, "@Time", "09:30", "@Temp", T[09:30] ), SELECTCOLUMNS ( T, "@Time", "09:40", "@Temp", T[09:40] ), SELECTCOLUMNS ( T, "@Time", "09:50", "@Temp", T[09:50] ), SELECTCOLUMNS ( T, "@Time", "10:00", "@Temp", T[10:00] ), SELECTCOLUMNS ( T, "@Time", "10:10", "@Temp", T[10:10] ), SELECTCOLUMNS ( T, "@Time", "10:20", "@Temp", T[10:20] ), SELECTCOLUMNS ( T, "@Time", "10:30", "@Temp", T[10:30] ), SELECTCOLUMNS ( T, "@Time", "10:40", "@Temp", T[10:40] ), SELECTCOLUMNS ( T, "@Time", "10:50", "@Temp", T[10:50] ), SELECTCOLUMNS ( T, "@Time", "11:00", "@Temp", T[11:00] ), SELECTCOLUMNS ( T, "@Time", "11:10", "@Temp", T[11:10] ), SELECTCOLUMNS ( T, "@Time", "11:20", "@Temp", T[11:20] ), SELECTCOLUMNS ( T, "@Time", "11:30", "@Temp", T[11:30] ), SELECTCOLUMNS ( T, "@Time", "11:40", "@Temp", T[11:40] ), SELECTCOLUMNS ( T, "@Time", "11:50", "@Temp", T[11:50] ), SELECTCOLUMNS ( T, "@Time", "12:00", "@Temp", T[12:00] ), SELECTCOLUMNS ( T, "@Time", "12:10", "@Temp", T[12:10] ), SELECTCOLUMNS ( T, "@Time", "12:20", "@Temp", T[12:20] ), SELECTCOLUMNS ( T, "@Time", "12:30", "@Temp", T[12:30] ), SELECTCOLUMNS ( T, "@Time", "12:40", "@Temp", T[12:40] ), SELECTCOLUMNS ( T, "@Time", "12:50", "@Temp", T[12:50] ), SELECTCOLUMNS ( T, "@Time", "13:00", "@Temp", T[13:00] ), SELECTCOLUMNS ( T, "@Time", "13:10", "@Temp", T[13:10] ), SELECTCOLUMNS ( T, "@Time", "13:20", "@Temp", T[13:20] ), SELECTCOLUMNS ( T, "@Time", "13:30", "@Temp", T[13:30] ), SELECTCOLUMNS ( T, "@Time", "13:40", "@Temp", T[13:40] ), SELECTCOLUMNS ( T, "@Time", "13:50", "@Temp", T[13:50] ), SELECTCOLUMNS ( T, "@Time", "14:00", "@Temp", T[14:00] ), SELECTCOLUMNS ( T, "@Time", "14:10", "@Temp", T[14:10] ), SELECTCOLUMNS ( T, "@Time", "14:20", "@Temp", T[14:20] ), SELECTCOLUMNS ( T, "@Time", "14:30", "@Temp", T[14:30] ), SELECTCOLUMNS ( T, "@Time", "14:40", "@Temp", T[14:40] ), SELECTCOLUMNS ( T, "@Time", "14:50", "@Temp", T[14:50] ), SELECTCOLUMNS ( T, "@Time", "15:00", "@Temp", T[15:00] ), SELECTCOLUMNS ( T, "@Time", "15:10", "@Temp", T[15:10] ), SELECTCOLUMNS ( T, "@Time", "15:20", "@Temp", T[15:20] ), SELECTCOLUMNS ( T, "@Time", "15:30", "@Temp", T[15:30] ), SELECTCOLUMNS ( T, "@Time", "15:40", "@Temp", T[15:40] ), SELECTCOLUMNS ( T, "@Time", "15:50", "@Temp", T[15:50] ), SELECTCOLUMNS ( T, "@Time", "16:00", "@Temp", T[16:00] ), SELECTCOLUMNS ( T, "@Time", "16:10", "@Temp", T[16:10] ), SELECTCOLUMNS ( T, "@Time", "16:20", "@Temp", T[16:20] ), SELECTCOLUMNS ( T, "@Time", "16:30", "@Temp", T[16:30] ), SELECTCOLUMNS ( T, "@Time", "16:40", "@Temp", T[16:40] ), SELECTCOLUMNS ( T, "@Time", "16:50", "@Temp", T[16:50] ), SELECTCOLUMNS ( T, "@Time", "17:00", "@Temp", T[17:00] ), SELECTCOLUMNS ( T, "@Time", "17:10", "@Temp", T[17:10] ), SELECTCOLUMNS ( T, "@Time", "17:20", "@Temp", T[17:20] ), SELECTCOLUMNS ( T, "@Time", "17:30", "@Temp", T[17:30] ), SELECTCOLUMNS ( T, "@Time", "17:40", "@Temp", T[17:40] ), SELECTCOLUMNS ( T, "@Time", "17:50", "@Temp", T[17:50] ), SELECTCOLUMNS ( T, "@Time", "18:00", "@Temp", T[18:00] ), SELECTCOLUMNS ( T, "@Time", "18:10", "@Temp", T[18:10] ), SELECTCOLUMNS ( T, "@Time", "18:20", "@Temp", T[18:20] ), SELECTCOLUMNS ( T, "@Time", "18:30", "@Temp", T[18:30] ), SELECTCOLUMNS ( T, "@Time", "18:40", "@Temp", T[18:40] ), SELECTCOLUMNS ( T, "@Time", "18:50", "@Temp", T[18:50] ), SELECTCOLUMNS ( T, "@Time", "19:00", "@Temp", T[19:00] ), SELECTCOLUMNS ( T, "@Time", "19:10", "@Temp", T[19:10] ), SELECTCOLUMNS ( T, "@Time", "19:20", "@Temp", T[19:20] ), SELECTCOLUMNS ( T, "@Time", "19:30", "@Temp", T[19:30] ), SELECTCOLUMNS ( T, "@Time", "19:40", "@Temp", T[19:40] ), SELECTCOLUMNS ( T, "@Time", "19:50", "@Temp", T[19:50] ), SELECTCOLUMNS ( T, "@Time", "20:00", "@Temp", T[20:00] ), SELECTCOLUMNS ( T, "@Time", "20:10", "@Temp", T[20:10] ), SELECTCOLUMNS ( T, "@Time", "20:20", "@Temp", T[20:20] ), SELECTCOLUMNS ( T, "@Time", "20:30", "@Temp", T[20:30] ), SELECTCOLUMNS ( T, "@Time", "20:40", "@Temp", T[20:40] ), SELECTCOLUMNS ( T, "@Time", "20:50", "@Temp", T[20:50] ), SELECTCOLUMNS ( T, "@Time", "21:00", "@Temp", T[21:00] ), SELECTCOLUMNS ( T, "@Time", "21:10", "@Temp", T[21:10] ), SELECTCOLUMNS ( T, "@Time", "21:20", "@Temp", T[21:20] ), SELECTCOLUMNS ( T, "@Time", "21:30", "@Temp", T[21:30] ), SELECTCOLUMNS ( T, "@Time", "21:40", "@Temp", T[21:40] ), SELECTCOLUMNS ( T, "@Time", "21:50", "@Temp", T[21:50] ), SELECTCOLUMNS ( T, "@Time", "22:00", "@Temp", T[22:00] ), SELECTCOLUMNS ( T, "@Time", "22:10", "@Temp", T[22:10] ), SELECTCOLUMNS ( T, "@Time", "22:20", "@Temp", T[22:20] ), SELECTCOLUMNS ( T, "@Time", "22:30", "@Temp", T[22:30] ), SELECTCOLUMNS ( T, "@Time", "22:40", "@Temp", T[22:40] ), SELECTCOLUMNS ( T, "@Time", "22:50", "@Temp", T[22:50] ), SELECTCOLUMNS ( T, "@Time", "23:00", "@Temp", T[23:00] ), SELECTCOLUMNS ( T, "@Time", "23:10", "@Temp", T[23:10] ), SELECTCOLUMNS ( T, "@Time", "23:20", "@Temp", T[23:20] ), SELECTCOLUMNS ( T, "@Time", "23:30", "@Temp", T[23:30] ), SELECTCOLUMNS ( T, "@Time", "23:40", "@Temp", T[23:40] ), SELECTCOLUMNS ( T, "@Time", "23:50", "@Temp", T[23:50] ) ) VAR __Result = AVERAGEX ( __Unpivot__, [@Temp] ) RETURN __ResultAlexisOlson Right, this is simply MC Aggregations and/or DAX Unpivot that I mentioned in the first post. Kudos to you for coding it all out!
This one allows for double aggregation if a GROUPBY step is added. I'm saying because in the original real life problem the data was energy consumption values and the requirement was to get the average of the daily total consumption (the average of the sum). Thus by adding a GROUPBY step you have the flexibility to double aggregate as you like.
Despite the fact I was not looking for a hardcoded solution but that was never listed as part of the conditions neither was the double aggregation requirement. Therefore this solution shall be marked as acceptable solution.
The way I see it, i am 100% with AlexisOlson solution just because of the stellar performance since we are talking about a production environment. Hardcoding 144 columns is a breeze with chatgpt. But, kudos to Greg for a smart approach, converting to csv and parsing text is a great learning suggestion.
These text-parsing solutions are clever but really slow. These are the timings I got for just the 31 days of January 2024.
tamerj1 solution
Greg_Deckler's [Measure 5] solution:The hard-coded solution to this problem is super fast:
Avg Temp = DIVIDE ( SUMX ( T, T[00:00] + T[00:10] + T[00:20] + T[00:30] + T[00:40] + T[00:50] + T[01:00] + T[01:10] + T[01:20] + T[01:30] + T[01:40] + T[01:50] + T[02:00] + T[02:10] + T[02:20] + T[02:30] + T[02:40] + T[02:50] + T[03:00] + T[03:10] + T[03:20] + T[03:30] + T[03:40] + T[03:50] + T[04:00] + T[04:10] + T[04:20] + T[04:30] + T[04:40] + T[04:50] + T[05:00] + T[05:10] + T[05:20] + T[05:30] + T[05:40] + T[05:50] + T[06:00] + T[06:10] + T[06:20] + T[06:30] + T[06:40] + T[06:50] + T[07:00] + T[07:10] + T[07:20] + T[07:30] + T[07:40] + T[07:50] + T[08:00] + T[08:10] + T[08:20] + T[08:30] + T[08:40] + T[08:50] + T[09:00] + T[09:10] + T[09:20] + T[09:30] + T[09:40] + T[09:50] + T[10:00] + T[10:10] + T[10:20] + T[10:30] + T[10:40] + T[10:50] + T[11:00] + T[11:10] + T[11:20] + T[11:30] + T[11:40] + T[11:50] + T[12:00] + T[12:10] + T[12:20] + T[12:30] + T[12:40] + T[12:50] + T[13:00] + T[13:10] + T[13:20] + T[13:30] + T[13:40] + T[13:50] + T[14:00] + T[14:10] + T[14:20] + T[14:30] + T[14:40] + T[14:50] + T[15:00] + T[15:10] + T[15:20] + T[15:30] + T[15:40] + T[15:50] + T[16:00] + T[16:10] + T[16:20] + T[16:30] + T[16:40] + T[16:50] + T[17:00] + T[17:10] + T[17:20] + T[17:30] + T[17:40] + T[17:50] + T[18:00] + T[18:10] + T[18:20] + T[18:30] + T[18:40] + T[18:50] + T[19:00] + T[19:10] + T[19:20] + T[19:30] + T[19:40] + T[19:50] + T[20:00] + T[20:10] + T[20:20] + T[20:30] + T[20:40] + T[20:50] + T[21:00] + T[21:10] + T[21:20] + T[21:30] + T[21:40] + T[21:50] + T[22:00] + T[22:10] + T[22:20] + T[22:30] + T[22:40] + T[22:50] + T[23:00] + T[23:10] + T[23:20] + T[23:30] + T[23:40] + T[23:50] ), 144 * COUNTROWS ( T ) )This doesn't work if you want a distinct count aggregation but SUM and MIN/MAX work just fine.
tamerj1 So, to be clear, we are not supposed to transform the table in any way with Power Query? At first glance, it looks like MC Aggregations. Multi-Column Aggregations (MC Aggregations) - Microsoft Fabric Community or maybe DAX Unpivot. DAX Unpivot - Microsoft Fabric Community
I think your other solution makes more sense. To unpivot 144 columns manually using DAX is not expected to be a pleasant experience. Unless you insist to proof the opposite and produce the required DAX over the provided sample data 😅
tamerj1 Here's another DAX approach that seems better than the other two I listed:
Measure = VAR __Text = SUBSTITUTE( TOCSV('T_MEASURES',,,0), ",", "|" ) VAR __Table = ADDCOLUMNS( GENERATESERIES(3, 144, 1), "__Value", PATHITEM( __Text, [Value]) + 0 ) VAR __Result = AVERAGEX( __Table, [__Value] ) RETURN __ResultThis one looks fantastic! 😍 I would like to see it working in the chart. Have you tried it? Otherwise, I'll do tomorrow morning.
tamerj1 I did try it and it seems to work great. Maybe a tad slow I suppose. I'll attach PBIX below signature.
Yep. If Import is allowed rather than a live connection or DirectQuery, then unpivoting in Power Query is a great solution.
For future readers, here's my summary of BIstvan's approach: Connect to the Power BI semantic model as an Analysis Services database and then do whatever you like (like unpivot) in Power Query. Here's a sample query:
let Source = AnalysisServices.Database( "powerbi://api.powerbi.com/v1.0/myorg/WorkspaceName", "Power Bi Challenge - Round2", [Query = "EVALUATE T_MEASURES"] ) in SourceIn the above, WorkspaceName should be the name of the workspace where the semantic model is stored, "Power BI Challenge - Round2" is the name of the dataset/semantic model, and Query is a simple EVALUATE statement to load the T_MEASURES table.
Greg_Deckler
I cannot thank you enough for the great input that you have provided in this subject. you really did a fantastic job that made more than happy and enjoyed realing every single paragraph and every single line of code. I personally learned much more than expected. I hope others did/will do.
Here is my original solution of the probelm with the explanationAverage Temperature = AVERAGEX ( T_MEASURES, VAR T1 = CALCULATETABLE ( T_MEASURES ) VAR String1 = TOCSV ( T1, 1, ",", TRUE ) VAR Items1 = SUBSTITUTE ( String1, UNICHAR ( 10 ), "|" ) VAR T2 = SELECTCOLUMNS ( { ( 1, 2 ) }, "@Headers", PATHITEM ( Items1, 1 ), "@Details", PATHITEM ( Items1, 2 ) ) VAR T3 = GENERATE ( T2, VAR HeaderString = [@Headers] VAR HeaderItems = SUBSTITUTE ( HeaderString, ",", "|" ) VAR DetailString = [@Details] VAR DetailItems = SUBSTITUTE ( DetailString, ",", "|" ) VAR Length = PATHLENGTH ( HeaderItems ) VAR T4 = GENERATESERIES ( 1, Length, 1 ) RETURN FILTER ( SELECTCOLUMNS ( T4, "@Header", PATHITEM ( HeaderItems, [Value] ), "@Detail", PATHITEM ( DetailItems, [Value] ) ), CONTAINSSTRING ( [@Header], ":" ) ) ) RETURN AVERAGEX ( T3, VALUE ( [@Detail] ) ) )tamerj1 Very cool, a lot of the same concepts and patterns! I need to review this solution in the file!
Anonymous I thought about an alternate solution instead of DAX.
@tamerj1 I have though about an alternate approach to solve the above challenge.
As in the original fole we do not have the option to model the data then we publish it and we set the maximum amount of refresh's possible(depending on licence)
We open a new .pbix file and then through XMLA endpoint we bring the published data in the new .pbix file. Where we unpivot all the columns with the values so we get rid of the multi column aggregations. From that on it pretty simple to build the visuals.
I have cheated a bit as it has not been solved with DAX but it works.I am also attaching a video link of the steps how to do it.
This is absolutely amazing!!! I didn't know you can do that!! Actually I was hoping to receive this type of solutions and you just made my wish come true. Thank you so much for you participation.
I will add all the accepted solutions to the original post along with links and attachement. I might also post the same in LinkedIn. I hope you don't mind tagging you and including your solution?
