topic Re: Simple Linear Regression in Quick Measures Gallery

Simple Linear Regression

Daniil — Fri, 13 Apr 2018 06:19:22 GMT

This measure allows you to predict dependent values Y from independent values X.

NAME:

Simple linear regression

DESCRIPTION:

Estimate Y values based on X values.

PARAMETERS:

Name: Category

Tooltip: The category for which you have known X and Y values

Type: Categorical field

Name: Measure X

Tooltip: Known X (independent) values

Type: Numerical field / measure

Name: Measure Y

Tooltip: Known Y (dependent) values

Type: Numerical field / measure

DAX:

Estimated {Measure Y} =
VAR Known =
    FILTER (
        SELECTCOLUMNS (
            ALLSELECTED ( {Category} ),
            "Known[X]", CALCULATE ( {Measure X} ),
            "Known[Y]", CALCULATE ( {Measure Y} )
        ),
        AND (
            NOT ( ISBLANK ( Known[X] ) ),
            NOT ( ISBLANK ( Known[Y] ) )
        )
    )
VAR Count_Items =
    COUNTROWS ( Known )
VAR Sum_X =
    SUMX ( Known, Known[X] )
VAR Sum_X2 =
    SUMX ( Known, Known[X] ^ 2 )
VAR Sum_Y =
    SUMX ( Known, Known[Y] )
VAR Sum_XY =
    SUMX ( Known, Known[X] * Known[Y] )
VAR Average_X =
    AVERAGEX ( Known, Known[X] )
VAR Average_Y =
    AVERAGEX ( Known, Known[Y] )
VAR Slope =
    DIVIDE (
        Count_Items * Sum_XY - Sum_X * Sum_Y,
        Count_Items * Sum_X2 - Sum_X ^ 2
    )
VAR Intercept =
    Average_Y - Slope * Average_X
RETURN
    Intercept + Slope * {Measure X}

For more details on and other uses of this quick measure, see my blog post on the subject:

https://xxlbi.com/blog/simple-linear-regression-in-dax/

eyJrIjoiZWNiNTQ2MGEtMjhlNS00YTNhLWE2NTktZDg3MzcxMjQ0NDc4IiwidCI6ImQzMmNkYzNmLTY1NTUtNGNhYy1iYjFhLTg2OWZiMTE0MzRlNSJ9

Re: Simple Linear Regression

tonymaclaren — Mon, 27 Nov 2017 13:16:15 GMT

I am using your Simple Linear Regression measure with great success. Thank You.

I am doing temperature studies and I want to find the first and last values of your measure in a given time interval which is filtered by a slicer. By doing this I can calculate the difference in the trend line over the interval of filtered years.

I can find the first and last year of the filtered interval by using FIRSTNONBLANK and LASTNONBLANK but I can't do this with your code because FIRSTNONBLANK and LASTNONBLANK only accept a column as an argument. I cannot seem to convert your measure into a calculated column to do this. Is there a version of your measure that produces a column? My data has two columns year and temperature. X and Y.

Any help would be much appreciated.

Tony Maclaren

Re: Simple Linear Regression

Daniil — Tue, 28 Nov 2017 08:59:14 GMT

Thanks for your feedback, Tony!

If I understand you correctly, there are a few ways to achieve your goal.

Here is an example of a measure:

Starting Temperature =
VAR Known =
    FILTER (
        SELECTCOLUMNS (
            ALLSELECTED ( 'Table'[Year] ),
            "Known[X]", 'Table'[Year] ),
            "Known[Y]", [Temperature]
        ),
        AND (
            NOT ( ISBLANK ( Known[X] ) ),
            NOT ( ISBLANK ( Known[Y] ) )
        )
    )
VAR First =
    TOPN ( 1, Known, Known[X], ASC )
RETURN
    MINX ( First, Known[Y] )

Re: Simple Linear Regression

tonymaclaren — Tue, 28 Nov 2017 19:35:08 GMT

Dear Daniil

Thank you for your very prompt response. I will be in Sydney, where I grew up, in early December!

Here is the tweaked code below (minus some brackets) that does the trick.Thank you so much.

This is an incredibly useful companion measure to your Simple Linear Regression measure, and because it works with measures it will find the equivalent of FIRSTNONBLANK in a measure. However it does not work correctly in the filtered context of a YEAR slider to filter the interval of years which the Simple Linear Regression measure does .The value changes with the slider, but the results do not correctly match the estimated starting value of the Estimated measure except for the first value of the whole dataset.

Starting Temperature =
VAR Known =
    FILTER (
        SELECTCOLUMNS (
            ALLSELECTED ( 'cetdata'[YEAR] ),
            "Known[X]", 'cetdata'[YEAR],
            "Known[Y]", [Estimated]
        ),
        AND (
            NOT ( ISBLANK ( Known[X] ) ),
            NOT ( ISBLANK ( Known[Y] ) )
        )
    )
VAR First =
    TOPN ( 1, Known, Known[X], ASC )
RETURN
    MINX ( First, Known[Y] )
            NOT ( ISBLANK ( Known[Y] ) )
        )
    )
VAR First =
    TOPN ( 1, Known, Known[X], ASC )
RETURN
    MINX ( First, Known[Y] )

Now we have to fix the filter context and figure out how to find [Ending temperature] so we can subtract [Starting Temperature] and get the trend change per filtered interval.

Thanks again--great stuff. No one else had a clue what my problem was or how to solve it.

Tony

Re: Simple Linear Regression

Daniil — Wed, 29 Nov 2017 03:24:32 GMT

Tony, you should visit the local Power BI User Group if you get a chance :-) The next meeting date should be announced at Meetup soon.

Try the following measures:

Starting Temperature = 
VAR Estimate =
    SELECTCOLUMNS (
        KEEPFILTERS ( ALLSELECTED ( 'cetdata'[YEAR] ) ),
        "Estimate[X]", 'cetdata'[YEAR],
        "Estimate[Y]", [Estimated]
    )
VAR First =
    TOPN ( 1, Estimate, Estimate[X], ASC )
RETURN
    MINX ( First, Estimate[Y] )

Ending Temperature = 
VAR Estimate =
    SELECTCOLUMNS (
        KEEPFILTERS ( ALLSELECTED ( 'cetdata'[YEAR] ) ),
        "Estimate[X]", 'cetdata'[YEAR],
        "Estimate[Y]", [Estimated]
    )
VAR Last =
    TOPN ( 1, Estimate, Estimate[X], DESC )
RETURN
    MAXX ( Last, Estimate[Y] )

Temperature Difference = [Ending Temperature] - [Starting Temperature]

Re: Simple Linear Regression

tonymaclaren — Wed, 29 Nov 2017 13:43:29 GMT

Dear Daniil

Thank you very much--all working perfectly now. KEEPFILTERS did the trick.I have really learned a lot! I have now ordered the SQLBI book.

One strange thing however,The [Estimated] measure which is your Simple linear Regression measure produces a total if displayed in a table. I do not quite understand this.

Best Regards

and thanks again

Tony

Re: Simple Linear Regression

Anonymous — Fri, 01 Dec 2017 12:45:12 GMT

Really excellent stuff !!

I had managed to do a linear regression line previously but this is so much cleaner

I did have to tweak yours slightly

1) Instead of a plain old Allselected i needed to

CALCULATETABLE (
SUMMARIZE (
FeederHistory,
FeederHistory[EventDateHour],
FeederHistory[EventDate]
),
ALLSELECTED ( FeederHistory )
),

and for some reason SSRS moaned at me for using SELCTEDVALUE so I replaced it with the older hasonevalue values paradigm

RETURN
Intercept
+ Slope
* IF (
HASONEVALUE ( FeederHistory[EventDate] ),
VALUES ( FeederHistory[EventDate] ))

Re: Simple Linear Regression

Anonymous — Fri, 01 Dec 2017 12:48:43 GMT

Really excellent stuff !!

I had managed to do a linear regression line previously but this is so much cleaner

I did have to tweak yours slightly

1) Instead of a plain old Allselected i needed to

CALCULATETABLE (
SUMMARIZE (
FeederHistory,
FeederHistory[EventDateHour],
FeederHistory[EventDate]
),
ALLSELECTED ( FeederHistory )
),

and for some reason SSRS moaned at me for using SELCTEDVALUE so I replaced it with the older hasonevalue values paradigm

RETURN
Intercept
+ Slope
* IF (
HASONEVALUE ( FeederHistory[EventDate] ),
VALUES ( FeederHistory[EventDate] ))

Re: Simple Linear Regression

Anonymous — Fri, 01 Dec 2017 12:51:56 GMT

I know this is a Power Bi portal but I had to make it work for both PBI and SSRS

Re: Simple Linear Regression

tonymaclaren — Fri, 01 Dec 2017 17:04:14 GMT

Thanks for your comments. If, like me you ever need to get the first and last values for a given filtered interval to calculate the total trend change over time, or rate of trend change over time, the Starting and Ending measures are great. When you get something like this working it really puts a smile on your face! With these complicated measures I can highly recomment DAX formatter if you are not already using it.

http://www.daxformatter.com/

Tony

Re: Simple Linear Regression

Daniil — Sat, 02 Dec 2017 03:56:31 GMT

tonymaclaren, it is likely that at the grand total level what you see is not a total, but the Intercept. This is because the following expression evaluates to Intercept:

Intercept + Slope * SELECTEDVALUE ( 'cetdata'[YEAR] )

At the grant total level, there is usually more than one 'cetdata'[YEAR], hence SELECTEDVALUE ( 'cetdata'[YEAR] ) returns BLANK, turning the multiplication into BLANK as well, leaving only Intercept.

I updated my blog post to deal with this kind of situations.

Anonymous, glad you found this useful!

Re: Simple Linear Regression

tonymaclaren — Sat, 02 Dec 2017 10:26:30 GMT

Dar Daniil

Thanks for the explanation.The same value also appears at the top of the [Estimated] measure, but only if the whole of the data for ('Ctedata'[YEAR])is displayed. is there a modification to eliminate this?

Tony

Re: Simple Linear Regression

Daniil — Sat, 02 Dec 2017 11:20:05 GMT

Tony, it's a bit difficult to advise anything without seeing your report -- can you share a sample of your data?

Re: Simple Linear Regression

tonymaclaren — Sat, 02 Dec 2017 12:27:07 GMT

Hi Daniil

Sure, no problem.What would you like me to supply and how.The full dataset is for 391 years.

Tony

Re: Simple Linear Regression

Daniil — Sat, 02 Dec 2017 13:34:48 GMT

Wow, 391 years of data sounds exciting :-) If you could reduce your data to the minimum amount with which you could reproduce your formula problems, that would be great.

Re: Simple Linear Regression

tonymaclaren — Sat, 02 Dec 2017 15:18:19 GMT

Thanks Daniil

I have attached a subset of the file that exhibits the exact same problem using the following measure:

Estimated tony Data =
VAR Known =
    FILTER (
        SELECTCOLUMNS (
            ALLSELECTED ( 'Tony_data'[YEAR] ),
            "Known[X]", 'Tony_data'[YEAR],
            "Known[Y]", [Averagetemp]
        ),
        AND (
            NOT ( ISBLANK ( Known[X] ) ),
            NOT ( ISBLANK ( Known[Y] ) )
        )
    )
VAR Count_Items =
    COUNTROWS ( Known )
VAR Sum_X =
    SUMX ( Known, Known[X] )
VAR Sum_X2 =
    SUMX ( Known, Known[X] ^ 2 )
VAR Sum_Y =
    SUMX ( Known, Known[Y] )
VAR Sum_XY =
    SUMX ( Known, Known[X] * Known[Y] )
VAR Average_X =
    AVERAGEX ( Known, Known[X] )
VAR Average_Y =
    AVERAGEX ( Known, Known[Y] )
VAR Slope =
    DIVIDE (
        Count_Items * Sum_XY - Sum_X * Sum_Y,
        Count_Items * Sum_X2 - Sum_X ^ 2
    )
VAR Intercept =
    Average_Y - Slope * Average_X
RETURN
    Intercept + Slope * SELECTEDVALUE ( 'Tony_data'[YEAR])

Averagetemp = SUM(Tony_data[AverageTemperature])

Tony

Re: Simple Linear Regression

Daniil — Sat, 02 Dec 2017 21:56:16 GMT

Tony, I can't reproduce the problem, unfortunately. Can you please attach a pbix file and screenshots with the problem highlighted?

Also, have you followed my updated blog post? I don't use SELECTEDVALUE anymore: https://xxlbi.com/blog/simple-linear-regression-in-dax/

Re: Simple Linear Regression

tonymaclaren — Sun, 03 Dec 2017 16:57:42 GMT

Hi Daniil

Thanks again for all your help. I have attached the whole file because it might interest you anyway. I sholuld stress it is a work in progress and my first use of Power BI. Please keep it to yourself.

The page labeled Tony Data with the table in it shows the problem clearly--although it does not reproduce in the chart on that page. It is not really a problem, but it would be nice for us to understand the behaviour.Play around with the date slider and you will notice that if you set the start date to 1960 only you see -32.72 at the top and bottom of the [Estimated Tony Data] column. Other start dates do not exhibit this problem.

I will take a look at the blog.

Tony

Re: Simple Linear Regression

leanh — Wed, 25 Apr 2018 14:41:23 GMT

Aye Tony, hope my suggestion could work for you with such problem:

You can try replacing this part:

RETURN
Intercept + Slope * SELECTEDVALUE ( 'Tony_data'[YEAR])

with this one:

      RETURN
SUMX( Tony_data ,
                Intercept + Slope * SELECTEDVALUE ( 'Tony_data'[YEAR])
            )

Btw, this one is the very first reply of mine in this place, so HELLO WORLD then :'>

Re: Simple Linear Regression

jwjwjwjwjw — Tue, 08 May 2018 15:00:19 GMT

Thanks for the measure Daniil, it is extremely helpful.

I am attempting to apply it to a forecasting problem that is relatively simple in excel, but is causing some trouble for me with Dax/PowerBI. I have 19 different categories that I need to run the regression on, and then I need to sum those categories for use on further regressions. Could your measure be modified to accomplish this or do I need to create 19 different measures?

Thanks for any help you could provide.