Forum Discussion
Correlation between same column, different items for time periods
- 8 years ago
In this sort of "pairwise comparison" scenario, where you have multiple entities in the same table (in this case distinguished by StockSymbolCurrency) I would normally follow the below steps.
My modifed copy of your pbix is here:
https://www.dropbox.com/s/y9h2ncitj46ee6a/PowerBiForumExample2%20Owen%20edit.pbix?dl=0- Create a copy of the entity dimension table, in your case a copy of ReferenceTable which I would call ReferenceTableComparison
- Create a relationship between StockBarDatExample and ReferenceTableComparison, but make it inactive
- Create the appropriate value measure that will be used for the company selected in ReferenceTable. For testing purposes I created
Average Close = AVERAGE ( StockBarDatExample[close] )
- Create the same measure for the Comparison Company, which activates the inactive relationship, and clears the filter on ReferenceTable:
Average Close Comparison = CALCULATE ( [Average Close], ALL ( ReferenceTable ), USERELATIONSHIP ( StockBarDatExample[StockSymbolCurrency], ReferenceTableComparison[StockSymbolCurrency] ) ) - You can then selected Company & Comparison Company using slicers, and use Average Close & Average Close Comparison together in visuals.
- The Pearson Correlation Coefficient can be calculated using a method similar to that used here. This relies on having the above two measures set up.
Here is the measure I tested with your data:Pearson Correlation Coefficient = VAR DateTimes = // Create a table of date/times where both stocks have a Close value FILTER ( SUMMARIZE ( StockBarDatExample, DateTable[DateKey], TimeTable[Column1] ), // Date & Time columns AND ( NOT ( ISBLANK ( [Average Close] ) ), NOT ( ISBLANK ( [Average Close Comparison] ) ) ) ) // Construct table of pairs of Close values VAR Known = SELECTCOLUMNS ( DateTimes, "Known[X]", [Average Close], "Known[Y]", [Average Close Comparison] ) // Calculate correlation coefficient VAR Count_Items = COUNTROWS ( Known ) VAR Average_X = AVERAGEX ( Known, Known[X] ) VAR Average_X2 = AVERAGEX ( Known, Known[X] ^ 2 ) VAR Average_Y = AVERAGEX ( Known, Known[Y] ) VAR Average_Y2 = AVERAGEX ( Known, Known[Y] ^ 2 ) VAR Average_XY = AVERAGEX ( Known, Known[X] * Known[Y] ) VAR CorrelationCoefficient = DIVIDE ( Average_XY - Average_X * Average_Y, SQRT ( ( Average_X2 - Average_X ^ 2 ) * ( Average_Y2 - Average_Y ^ 2 ) ) ) RETURN CorrelationCoefficient
The test report page in the above pbix looks like this:
Hopefully that helps. :)
Regards,
Owen
Hi OwenAuger,
Thank you for posting this dashboard!
I am trying to do something similar with multiple stocks. Your appraoch looks promising however I think the result you are getting is not correct in your example I believe the correlation should be 0.97714 instead of 0.94. It seems to be linked to the issue discussed in this thread
Thanks for your reply on this topic 🙂
My original reply created a measure that calculation the correlation coefficient based only on records where X and Y are both nonblank.
With that assumption, I believe the correlation coefficient of 0.94 is correct (can be tested by pasting data to Excel, removing rows where either X or Y is blank, and applying the CORREL function).
For your particular case, I'm not sure where the calculation is going wrong. It could possibly relate to how the Known table is constructed. Known should include only the rows to be included in the correlation coefficient calculation.
Could you post some more detail, even dummy data in a PBIX that illustrates when the calculation doesn't produce the expected result?
Regards,
Owen