Forum Discussion

Mohan128256's avatar
Mohan128256
Helper IV
1 year ago

Optimize complex dax measure

Hello All,

After going through all the possible ways of optimization of a measure which i have written, I am reaching out to you community members to help me out.

 

below is dax query which i have written to get the new customer reference name if there are any changes happend in within the date range selected based on sales order id, product id and line creation number.

 

Customer Reference (H) New = 
VAR _MinDate =
    CALCULATETABLE(
        ADDCOLUMNS(
            SUMMARIZE(
                'Sales Orders Snapshots',
                'Sales Orders Snapshots'[Sales Order ID (H)],
                'Sales Orders Snapshots'[Product ID (L)],
                'Sales Orders Snapshots'[Line Creation Sequence Number (L)]
            ),
            "@SnapDate", Max( 'Sales Orders Snapshots'[SnapDate] )
        ),
        ALLSELECTED()
    )
VAR _FilterSnap =
    TREATAS(
        _MinDate,
        'Sales Orders Snapshots'[Sales Order ID (H)],
        'Sales Orders Snapshots'[Product ID (L)],
        'Sales Orders Snapshots'[Line Creation Sequence Number (L)],
        'Sales Orders Snapshots'[SnapDate]
    )
VAR _Result =
    CALCULATE(
        MAX( 'Sales Orders Snapshots'[Customer Reference (H)]),
        ALLEXCEPT(
            'Sales Orders Snapshots',
            'Sales Orders Snapshots'[Sales Order ID (H)],
            'Sales Orders Snapshots'[Product ID (L)],
            'Sales Orders Snapshots'[Line Creation Sequence Number (L)]
        ),
        KEEPFILTERS( _FilterSnap )
    )
RETURN _Result

 this measure gives me right value but when i see the Server engine and formula engine timings it still bothering me as it is taking 18+seconds.

 

DAX Query from performance analyzer

// DAX Query
DEFINE
	VAR __DS0Core = 
		SUMMARIZECOLUMNS(
			'Sales Orders Snapshots'[Sales Order ID (H)],
			'Sales Orders Snapshots'[Product ID (L)],
			'Sales Orders Snapshots'[MGA Original Customer ID (H)],
			'Sales Orders Snapshots'[MGA Original Customer Name],
			'Sales Orders Snapshots'[Original Legal Entity (L)],
			'Sales Orders Snapshots'[Order Created Date (H)],
			'Sales Orders Snapshots'[Original Region Name (H)],
			'Sales Orders Snapshots'[Original Sales Entity Name (H)],
			'Sales Orders Snapshots'[Original Forecast Responsible Group Name (H)],
			'Sales Orders Snapshots'[Original SPA Name (H)],
			'Sales Orders Snapshots'[Sales Responsible Name (H)],
			'Sales Orders Snapshots'[D365 SO Link],
			'Sales Orders Snapshots'[Product Name],
			'Sales Orders Snapshots'[Line Creation Sequence Number (L)],
			'Sales Orders Snapshots'[Line Created Date (L)],
			'Sales Orders Snapshots'[MGA Original Order ID (L)],
			"Customer_Reference__H__New", 'Sales Orders Snapshots'[Customer Reference (H) New]
		)

	VAR __DS0PrimaryWindowed = 
		TOPN(
			501,
			__DS0Core,
			[Customer_Reference__H__New],
			0,
			'Sales Orders Snapshots'[Sales Order ID (H)],
			1,
			'Sales Orders Snapshots'[Product ID (L)],
			1,
			'Sales Orders Snapshots'[MGA Original Customer ID (H)],
			1,
			'Sales Orders Snapshots'[MGA Original Customer Name],
			1,
			'Sales Orders Snapshots'[Original Legal Entity (L)],
			1,
			'Sales Orders Snapshots'[Order Created Date (H)],
			1,
			'Sales Orders Snapshots'[Original Region Name (H)],
			1,
			'Sales Orders Snapshots'[Original Sales Entity Name (H)],
			1,
			'Sales Orders Snapshots'[Original Forecast Responsible Group Name (H)],
			1,
			'Sales Orders Snapshots'[Original SPA Name (H)],
			1,
			'Sales Orders Snapshots'[Sales Responsible Name (H)],
			1,
			'Sales Orders Snapshots'[D365 SO Link],
			1,
			'Sales Orders Snapshots'[Product Name],
			1,
			'Sales Orders Snapshots'[Line Creation Sequence Number (L)],
			1,
			'Sales Orders Snapshots'[Line Created Date (L)],
			1,
			'Sales Orders Snapshots'[MGA Original Order ID (L)],
			1
		)

EVALUATE
	__DS0PrimaryWindowed

ORDER BY
	[Customer_Reference__H__New] DESC,
	'Sales Orders Snapshots'[Sales Order ID (H)],
	'Sales Orders Snapshots'[Product ID (L)],
	'Sales Orders Snapshots'[MGA Original Customer ID (H)],
	'Sales Orders Snapshots'[MGA Original Customer Name],
	'Sales Orders Snapshots'[Original Legal Entity (L)],
	'Sales Orders Snapshots'[Order Created Date (H)],
	'Sales Orders Snapshots'[Original Region Name (H)],
	'Sales Orders Snapshots'[Original Sales Entity Name (H)],
	'Sales Orders Snapshots'[Original Forecast Responsible Group Name (H)],
	'Sales Orders Snapshots'[Original SPA Name (H)],
	'Sales Orders Snapshots'[Sales Responsible Name (H)],
	'Sales Orders Snapshots'[D365 SO Link],
	'Sales Orders Snapshots'[Product Name],
	'Sales Orders Snapshots'[Line Creation Sequence Number (L)],
	'Sales Orders Snapshots'[Line Created Date (L)],
	'Sales Orders Snapshots'[MGA Original Order ID (L)]

 

Server Timings:

 

Server engine as 3 scans:

Scan1 Query:

 

SELECT 'Sales Orders Snapshots'[Sales Order ID ( H ) ], 'Sales Orders Snapshots'[Line Creation Sequence Number ( L ) ], 'Sales Orders Snapshots'[Product ID ( L ) ] FROM 'Sales Orders Snapshots';   

 

Scan2 Query: 

SELECT 'Sales Orders Snapshots'[Sales Order ID ( H ) ], 'Sales Orders Snapshots'[Line Creation Sequence Number ( L ) ], 'Sales Orders Snapshots'[Product ID ( L ) ], MAX ( 'MinMaxColumnPositionCallback' ( PFDATAID ( 'Sales Orders Snapshots'[Customer Reference ( H ) ] ) ) ) FROM 'Sales Orders Snapshots' WHERE  ( 'Sales Orders Snapshots'[Sales Order ID ( H ) ], 'Sales Orders Snapshots'[Product ID ( L ) ], 'Sales Orders Snapshots'[Line Creation Sequence Number ( L ) ], 'Sales Orders Snapshots'[SnapDate] ) IN { ( '100S1040554', '656040-M8', '1', 45553.000000 ) , ( '100S0587644', '662737-000', '1', 45553.000000 ) , ( '411S0083387', '651489-E7', '4', 45553.000000 ) , ( '411S0094134', '593140-RF', '4', 45553.000000 ) , ( '100100S1115042', '541301-C3', '1', 45553.000000 ) , ( '100S0987039', '514527-CBULK', '7', 45553.000000 ) , ( '100S0112135', '618338-M', '1', 45553.000000 ) , ( '100S0331324', '587187-C3', '7', 45553.000000 ) , ( '100100S0377131', '587378-EUC', '39', 45553.000000 ) , ( '100S0599708', '591856-X2EUCALT', '3', 45553.000000 ) ..[20,49,990 total tuples, not all displayed]};   Estimated size: rows = 20,47,054  bytes = 3,27,52,864

Scan3 Query:

SELECT 'Sales Orders Snapshots'[D365 SO Link], 'Sales Orders Snapshots'[Product Name], 'Sales Orders Snapshots'[Sales Order ID ( H ) ], 'Sales Orders Snapshots'[Line Creation Sequence Number ( L ) ], 'Sales Orders Snapshots'[Product ID ( L ) ], 'Sales Orders Snapshots'[MGA Original Customer ID ( H ) ], 'Sales Orders Snapshots'[MGA Original Customer Name], 'Sales Orders Snapshots'[Original Forecast Responsible Group Name ( H ) ], 'Sales Orders Snapshots'[Original Region Name ( H ) ], 'Sales Orders Snapshots'[Original Sales Entity Name ( H ) ], 'Sales Orders Snapshots'[Original SPA Name ( H ) ], 'Sales Orders Snapshots'[Order Created Date ( H ) ], 'Sales Orders Snapshots'[Original Legal Entity ( L ) ], 'Sales Orders Snapshots'[Sales Responsible Name ( H ) ], 'Sales Orders Snapshots'[MGA Original Order ID ( L ) ], 'Sales Orders Snapshots'[Line Created Date ( L ) ] FROM 'Sales Orders Snapshots';   Estimated size: rows = 20,49,991  bytes = 13,11,99,424

 

Any suggestions or changes over the measure which i have written.

Please help.

 

Thanks,

Mohan V.

 

15 Replies

  • Mohan128256 It will be easier if you share pbix file using one drive/google drive with the expected output. Remove any sensitive information before sharing.

  • Look at the query plan and find rows with excessive number of records. That's where the cartesians happen that you need to eliminate.

     

    Break your measure down into smaller parts and evaluate these part by part to see where it becomes expensive.

    • lbendlin's avatar
      lbendlin
      Super User

       

       

      There are too many columns in that table visual.  Anytime you see a horizontal scrollbar in a visual you know you have too many columns.

       

      Your _minDate filter is not filtering much. You seemingly spend over 4 seconds to recalculate the same SnapDate over 250000 rows.

      CALCULATETABLE(
              ADDCOLUMNS(
                  SUMMARIZE(
                      'Sales Orders Snapshots',
                      'Sales Orders Snapshots'[Sales Order ID (H)],
                      'Sales Orders Snapshots'[Product ID (L)],
                      'Sales Orders Snapshots'[Line Creation Sequence Number (L)]
                  ),
                  "@SnapDate", MIN( 'Sales Orders Snapshots'[SnapDate] )
              ),
              ALLSELECTED()
          )
       
      And then you throw ALLEXCEPT, KEEPFILTERS and TREATAS all into a pile
       
      VAR _Result =
          CALCULATE(
              MAX( 'Sales Orders Snapshots'[Customer Item Number (L)]),
              ALLEXCEPT(
                  'Sales Orders Snapshots',
                  'Sales Orders Snapshots'[Sales Order ID (H)],
                  'Sales Orders Snapshots'[Product ID (L)],
                  'Sales Orders Snapshots'[Line Creation Sequence Number (L)]
              ),
              KEEPFILTERS( _FilterSnap )
          )
       
      I don't understand what you are trying to achieve with that. Maybe you can explain the business intent?
       
      As a side note - That file is way too big. Please provide sample data that fully covers your issue- but not more.. Do not include anything not related to the issue. Do not include any sensitive information.
      Please show the expected outcome based on the sample data you provided.
      • Mohan128256's avatar
        Mohan128256
        Helper IV

        lbendlin my heartfelt thanks and appreciate you on spending your personal time on this to help me out here.

         

        Here is what i am trying to do with the measures which i have created.

         

        Ex:- Customer Reference (H) column.

         

        As per the above screenshot,

        I have 30days of data for the salesorderid = 100S1077426, Productid = 594963-W, line creation number = 1.

        Customer Reference (H) New measure is used to calculate the latest value available in the timeperiod which we have filtered using date range slicer.

        In this case, for the max date 18-09-2024, we have customer reference value as 6631649637

         

        Customer Reference (H) Before measure is used to calculate the initial value available in the timeperiod which we have filtered using date range slicer.

        In this case, for the min date 20-08-2024, we have customer reference value as PENDING2

         

        So Customer Reference (H) New should show 6631649637, Customer Reference (H) Beforeshould show PENDING2

         

        But if i change the date range to 10-09-2024 to 18-09-2024 then both should show the same value, as the max date = 18-09-2024, min date=10-09-2024 has the same value.

        As you see, the measures which i have created are giving me the right result and they are working fine when i filter out the data with limited records.

        But actually, the table has 100+ millions of records, where I have to show the columns with their New and before values with conditional formatting if there is difference between New and before, hightlit it.

         

        I have limited the data.

        Please check the same file over the drive.

         

        Hope i did gave you the details which you require to look into this.

         

        Let me know if you need anymore details.

         

        Thanks,

        Mohan V.

         

  • lbendlin parry2k I have shared the file.. Request you to please have a look and help me out with the suggestions to make.

    highly appreciate your efforts on this.

     

    Thanks,

    Mohan V.

    • lbendlin's avatar
      lbendlin
      Super User

      Still way too big. (619 MB)

       

      Distill it down further.

      • Mohan128256's avatar
        Mohan128256
        Helper IV

        lbendlini have reduced to 1.3 mb with only 4 Salesorder id's data.
        Please check and let me know.