Microsoft Fabric Community Conference 2025, March 31 - April 2, Las Vegas, Nevada. Use code FABINSIDER for a $400 discount.
Register nowGet inspired! Check out the entries from the Power BI DataViz World Championships preliminary rounds and give kudos to your favorites. View the vizzies.
I'm using direct query and trying to do a count on a field in a visual. I expect this to not be super duper efficient, but for whatever reason it is extremely bad. I was able to look at the query in DPA that power BI is sending and it looks like this:
SELECT TOP (1000001) [semijoin4].[c28],[semijoin4] .[c81], (COUNT_BIG(DISTINCT [basetable2].[c217]) + MAX( CASE WHEN [basetable2].[c217] IS NULL THEN 1 ELSE 0 END)) AS [a0] FROM ( ( ( SELECT [t5].[EventListKey] AS [c98],[t5] .[BeginDateOnly] AS [c134],[t9] .[EventLabel] AS [c217] FROM ( ( SELECT [$Table].[EventListKey] as [EventListKey], [$Table] .[SourceSystem] as [SourceSystem], [$Table] .[SourceSystemID] as [SourceSystemID], [$Table] .[SourceSystemVersion] as [SourceSystemVersion], [$Table] .[ETLDatetime] as [ETLDatetime], [$Table] .[ETLVersion] as [ETLVersion], [$Table] .[RecordVersion] as [RecordVersion], [$Table] .[EventNum] as [EventNum], [$Table] .[EventStep] as [EventStep], [$Table] .[EventGroup] as [EventGroup], [$Table] .[EventLabel] as [EventLabel], [$Table] --I've removed some lines for the forum .[BeginDateOnly] as [BeginDateOnly], [$Table] .[BeginTimeOnly] as [BeginTimeOnly] FROM [UDM].[TableQualifiedEventList] as [$Table] ) AS [t9] LEFT OUTER JOIN ( SELECT [$Table].[EventListKey] as [EventListKey], [$Table] .[SourceSystem] as [SourceSystem], [$Table] .[SourceSystemID] as [SourceSystemID], [$Table] .[SourceSystemVersion] as [SourceSystemVersion], [$Table] .[ETLDatetime] as [ETLDatetime], [$Table] .[ETLVersion] as [ETLVersion], [$Table] .[RecordVersion] as [RecordVersion], [$Table] .[EventNum] as [EventNum], [$Table] .[EventStep] as [EventStep], [$Table] .[EventGroup] as [EventGroup], [$Table] .[EventLabel] as [EventLabel], [$Table] --I've removed some lines for the forum .[BeginDateOnly] as [BeginDateOnly], [$Table] .[BeginTimeOnly] as [BeginTimeOnly] FROM [UDM].[TableEventList] as [$Table] ) AS [t5] ON ( [t9].[EventListKey] = [t5].[EventListKey] ) ) WHERE ( ( ( [t9].[MetricType] = 'Sustained Event' ) AND ( [t9].[Normalization] = 'Normalized per IEEE 1366' ) ) AND ( [t9].[Qualification] = 'Qualified Outage' ) ) GROUP BY [t5].[EventListKey],[t5] .[BeginDateOnly],[t9] .[EventLabel] ) AS [basetable2] INNER JOIN ( ( SELECT 2014 AS [c28], CAST( '20140701 00:00:00' AS datetime) AS [c134] ) UNION ALL ( SELECT 2014 AS [c28], CAST( '20140702 00:00:00' AS datetime) AS [c134] ) UNION ALL ( SELECT 2014 AS [c28], CAST( '20140703 00:00:00' AS datetime) AS [c134] ) UNION ALL ( SELECT 2014 AS [c28], CAST( '20140704 00:00:00' AS datetime) AS [c134] ) UNION ALL ( SELECT 2014 AS [c28], CAST( '20140705 00:00:00' AS datetime) AS [c134] ) UNION ALL ( SELECT 2014 AS [c28], CAST( '20140706 00:00:00' AS datetime) AS [c134] ) UNION ALL ( SELECT 2014 AS [c28], CAST( '20140707 00:00:00' AS datetime) AS [c134] ) UNION ALL ( SELECT 2014 AS [c28], CAST( '20140708 00:00:00' AS datetime) AS [c134] ) UNION ALL ( SELECT 2014 AS [c28], CAST( '20140709 00:00:00' AS datetime) AS [c134] ) UNION ALL ( SELECT 2014 AS [c28], CAST( '20140710 00:00:00' AS datetime) AS [c134] ) UNION ALL ( SELECT 2014 AS [c28], CAST( '20140711 00:00:00' AS datetime) AS [c134] ) UNION ALL ( SELECT 2014 AS [c28], CAST( '20140712 00:00:00' AS datetime) AS [c134] ) UNION ALL ( SELECT 2014 AS [c28], CAST( '20140713 00:00:00' AS datetime) AS [c134] ) UNION ALL ( SELECT 2014 AS [c28], CAST( '20140714 00:00:00' AS datetime) AS [c134] ) UNION ALL ( SELECT 2014 AS [c28], CAST( '20140715 00:00:00' AS datetime) AS [c134] ) UNION ALL ( SELECT 2014 AS [c28], CAST( '20140716 00:00:00' AS datetime) AS [c134] ) UNION ALL ( SELECT 2014 AS [c28], CAST( '20140717 00:00:00' AS datetime) AS [c134] ) UNION ALL ( SELECT 2014 AS [c28], CAST( '2014071
The data in my file is basically Customer data, Customer - Event bridge table, and then Event Details. So basically my goal is to get a count of distinct event labels that meet certain criteria in that table, grouped by customer ID. This is how I would get the same information out of sql:
select PremiseID --CustomerID ,year(BeginDateOnly) ,count(EventLabel) Cnt from tablecustomerpremises CP join TableEventPremise EP On 1=1 and EP.CustomerPremiseKey = CP.CustomerPremiseKey and EP.SourceSystemID = 2 join TableQualifiedEventList EL on 1=1 and EL.EventListKey = EP.EventListKey and EL.SourceSystemID = 2 where 1=1 and EL.Qualification = 'Qualified' and CP.SourceSystemID = 2 group by premiseid ,year(BeginDateOnly) having count(EventLabel) > 3 order by year(BeginDateOnly) desc ,count(EventLabel) desc
How do I figure out why PBI is treating it this way? The only thing I can th ink of is that there is a date table that I am linking to on the QualifiedEventList that is technically in a different DB on the same server, but I set that table to Dual so that it shouldn't be going out to query that, but I also am not using any filters or sorting with the date table in the PBI report, so I don't know why it's doing this?
Looks like you want to group by year, which need a date, and your date is stored as string in the source. Looking at the SQL statement, you are also sorting the result, which is not needed since you can do this in the visual itself.
Could you try to group the results by LEFT(4) instead of year so the cast to date does not have to happen, and remove the sorting?
OK I've made some changes to make sure that everything is linking off of the date datatype, and now it's doing something similar by trying to pass the a list of the premiseIDs:
SELECT TOP (1000001) [semijoin1].[c81], (COUNT_BIG(DISTINCT [basetable0].[c177]) + MAX( CASE WHEN [basetable0].[c177] IS NULL THEN 1 ELSE 0 END)) AS [a0] FROM ( ( SELECT [t5].[EventListKey] AS [c98],[t8] .[EventLabel] AS [c177] FROM ( ( SELECT [$Table].[EventListKey] as [EventListKey], [$Table] .[SourceSystem] as [SourceSystem], [$Table] .[SourceSystemID] as [SourceSystemID], [$Table] --hiding for the forum .[Qualification] as [Qualification], [$Table] .[ResponsibleGroup] as [ResponsibleGroup], [$Table] .[BeginDateOnly] as [BeginDateOnly], [$Table] .[BeginTimeOnly] as [BeginTimeOnly] FROM [UDM].[TableQualifiedEventList] as [$Table] ) AS [t8] LEFT OUTER JOIN ( SELECT [$Table].[EventListKey] as [EventListKey], [$Table] .[SourceSystem] as [SourceSystem], [$Table] .[SourceSystemID] as [SourceSystemID], [$Table] .[SourceSystemVersion] as [SourceSystemVersion], [$Table] --hiding for the forum .[DispatchedOffice] as [DispatchedOffice], [$Table] .[BeginDateOnly] as [BeginDateOnly], [$Table] .[BeginTimeOnly] as [BeginTimeOnly] FROM [UDM].[TableEventList] as [$Table] ) AS [t5] ON ( [t8].[EventListKey] = [t5].[EventListKey] ) ) WHERE ( ( ( [t8].[MetricType] = 'Sustained Event' ) AND ( [t8].[Normalization] = 'Normalized' ) ) AND ( [t8].[Qualification] = 'Qualified Outage' ) ) GROUP BY [t5].[EventListKey],[t8] .[EventLabel] ) AS [basetable0] INNER JOIN ( SELECT [t4].[PremiseID] AS [c81],[t5] .[EventListKey] AS [c98] FROM ( ( ( ( SELECT [$Table].[EventPremiseKey] as [EventPremiseKey], [$Table] .[SourceSystem] as [SourceSystem], [$Table] .[SourceSystemID] as [SourceSystemID], [$Table] --hiding for the forum .[EventListKey] as [EventListKey] FROM [UDM].[TableEventPremise] as [$Table] ) AS [t6] LEFT OUTER JOIN ( SELECT [_].[CustomerPremiseKey], [_] .[SourceSystem], [_] .[SourceSystemID], [_] --hiding for the forum .[GPSLatitude], [_] .[GPSLongitude], [_] .[MeterID], [_] .[MeterFieldName], [_] .[ServiceBirth], [_] .[ServiceDeath], [_] .[PrimaryDeviceID], [_] .[PrimaryDeviceFieldLabel] FROM [UDM].[TableCustomerPremises] as [_] WHERE [_].[SourceSystemID] = 2 AND [_].[SourceSystemID] is not null ) AS [t4] ON ( [t6].[CustomerPremiseKey] = [t4].[CustomerPremiseKey] ) ) LEFT OUTER JOIN ( SELECT [$Table].[EventListKey] as [EventListKey], [$Table] .[SourceSystem] as [SourceSystem], [$Table] .[SourceSystemID] as [SourceSystemID], [$Table] .[SourceSystemVersion] as [SourceSystemVersion], [$Table] --hiding for the forum .[DispatchedOffice] as [DispatchedOffice], [$Table] .[BeginDateOnly] as [BeginDateOnly], [$Table] .[BeginTimeOnly] as [BeginTimeOnly] FROM [UDM].[TableEventList] as [$Table] ) AS [t5] ON ( [t6].[EventListKey] = [t5].[EventListKey] ) ) LEFT OUTER JOIN ( SELECT [$Table].[EventListKey] as [EventListKey], [$Table] .[SourceSystem] as [SourceSystem], [$Table] .[SourceSystemID] as [SourceSystemID], [$Table] --hiding for the forum .[BeginDateOnly] as [BeginDateOnly], [$Table] .[BeginTimeOnly] as [BeginTimeOnly] FROM [UDM].[TableQualifiedEventList] as [$Table] ) AS [t8] ON ( [t5].[EventListKey] = [t8].[EventListKey] ) ) WHERE ( ( ( [t8].[MetricType] = 'Sustained Event' ) AND ( [t8].[Normalization] = 'Normalized per IEEE 1366' ) ) AND ( ( [t8].[Qualification] = 'Qualified Outage' ) AND ( ([t4].[PremiseID] IN (1380036215,6710121583,3020456851,4070193589,3030641333,6950759772, 4360614969,9120736306,8970027451,4640296258,6720221414,3730218667,2230563108,6300668790, 9370555164,9950824118,4860938001,2050851077,5660076649,1950563439,5090377082,9150849585, 8770496798,7300857654,9510649132,9700166137,8610187412,4650946407,2730899016,6730178349, 490669915,9010594331,1680679614,6160132619,8490451331,9100200244,2540760976,2050727636, 40375263,4360651321,2900344154,6920503881,5160068541,250666575,1090901008,8180416967, 3620916542,3880290214,6600053663,5850434770,1860272155,2120411743,4540619244,3490664492, 8070893815,3890738177,5410104832,2060641227,8280941255,7180106175,2750674205,910691443, 5030440726,5770935062,5650776145,8410810901,9130167156,7440554718,8400005438,4210965680, 8270452750,3780133095,360770338,7910613420,3350080990,430462603,4290333969,7680711386,
I don't understand why it's doing this at all.
Did you read this article?
OK this looks a lot like the problem I'm running into, because it is trying to pass the initial table data as filter parameters to the table on the other side of the bridge. But I'm kind of confused though because all of the tables are sql and in the same DB etc. Also I'm not understanding where I need to put the code that guy recommends.
Thanks for your help!!!
OK I think I've finally tracked down the main culprit of this being the 'lastdate' and 'YTD' functions.
The main KPI i'm going for is based on two other components. One component is cumulative and adds up over time based on things that happen to the customers. The other component is the number of customers and is measured once a month. It is not 'cumulative' in nature, but we have to use the most recent count.
So here we go, number of customers is defined as:
Customer Count Center = CALCULATE( SUM('UDM TableServedByDate Center'[DistinctPremises]) ,LASTDATE('2 UDM DateTable'[CALENDAR_DATE]) ,'UDM TableServedByDate Center'[Service Center] in values('UDM TableGeoHierarchy'[Service Center]) )
I created a matrix visual to see what the value is over time and verify things are working. The query generated is massive, but there is a lot of this:
(SELECT @Param1 AS [c248], @Param2 AS [c245],CAST( '20130131 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param3 AS [c248], @Param4 AS [c245],CAST( '20130228 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param5 AS [c248], @Param6 AS [c245],CAST( '20130331 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param7 AS [c248], @Param8 AS [c245],CAST( '20130430 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param9 AS [c248], @Param10 AS [c245],CAST( '20130531 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param11 AS [c248], @Param12 AS [c245],CAST( '20130630 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param13 AS [c248], @Param14 AS [c245],CAST( '20130731 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param15 AS [c248], @Param16 AS [c245],CAST( '20130831 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param17 AS [c248], @Param18 AS [c245],CAST( '20130930 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param19 AS [c248], @Param20 AS [c245],CAST( '20131031 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param21 AS [c248], @Param22 AS [c245],CAST( '20131130 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param23 AS [c248], @Param24 AS [c245],CAST( '20131231 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param25 AS [c248], @Param26 AS [c245],CAST( '20140131 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param27 AS [c248], @Param28 AS [c245],CAST( '20140228 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param29 AS [c248], @Param30 AS [c245],CAST( '20140331 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param31 AS [c248], @Param32 AS [c245],CAST( '20140430 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param33 AS [c248], @Param34 AS [c245],CAST( '20140531 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param35 AS [c248], @Param36 AS [c245],CAST( '20140630 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param37 AS [c248], @Param38 AS [c245],CAST( '20140731 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param39 AS [c248], @Param40 AS [c245],CAST( '20140831 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param41 AS [c248], @Param42 AS [c245],CAST( '20140930 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param43 AS [c248], @Param44 AS [c245],CAST( '20141031 00:00:00' AS datetime) AS [c244] ) UNION ALL (SELECT @Param45 AS [c248], @Param46
It appears that for every row, it is generating the date as a string parameter and then trying to cast and pass that back to the sql server. So it's basically transmitting a boat load of data instead of using the connections in the model.
So my guess is that LASTDATE() is somehow the culprit. Reading on the link provided above, and other similar links, I see that query folding is not utilized if there is no equivalent funciton within the source (for me sql server). Pulled up LastDate function and lo and behold, the line I somehow didn't see until now:
This DAX function is not supported for use in DirectQuery mode. For more information about limitations in DirectQuery models, see https://go.microsoft.com/fwlink/?LinkId=219172.
Frack. So I need an alternative to LASTDATE. Anybody have suggestions?
So I tried to implement these
TOTALYTD ALT = CALCULATE ( SUM ( Sales[SalesAmount] ), FILTER ( ALL ( 'Calendar'[Date] ), 'Calendar'[Date] <= MAX ( 'Calendar'[Date] ) && YEAR ( 'Calendar'[Date] ) = YEAR ( MAX ( 'Calendar'[Date] ) ) ) )
LASTDATE ALT = CALCULATE ( <expression>, FILTER ( ALL ( table[column] ), table[column] = MAX ( table[column] ) ) )
And ended up with
Customer Count Center = CALCULATE( SUM('UDM TableServedByDate Center'[DistinctPremises]) ,filter( ALLSELECTED('UDM TableServedByDate Center'[CALENDAR_DATE]) ,'UDM TableServedByDate Center'[CALENDAR_DATE] = max('UDM TableServedByDate Center'[CALENDAR_DATE]) ) ,'UDM TableServedByDate Center'[Service Center] in values('UDM TableGeoHierarchy'[Service Center]) //,USERELATIONSHIP('UDM TableServedByDate Center'[Service Center],'UDM TableGeoHierarchy'[Service Center]) )
But it's still doing it in the query. The only thing I am doing here is loading a matrix visualization to show this measure Customer Count Center vs the Calendar_Date field as it exists in my calendar table.
(SELECT CAST( '20141201 00:00:00' AS datetime) AS [c198],CAST( '20141201 00:00:00' AS datetime) AS [c165] ) UNION ALL (SELECT CAST( '20140301 00:00:00' AS datetime) AS [c198],CAST( '20140301 00:00:00' AS datetime) AS [c165] ) UNION ALL (SELECT CAST( '20160701 00:00:00' AS datetime) AS [c198],CAST( '20160701 00:00:00' AS datetime) AS [c165] ) UNION ALL (SELECT CAST( '20170601 00:00:00' AS datetime) AS [c198],CAST( '20170601 00:00:00' AS datetime) AS [c165] ) UNION ALL (SELECT CAST( '20151101 00:00:00' AS datetime) AS [c198],CAST( '20151101 00:00:00' AS datetime) AS [c165] ) UNION ALL (SELECT CAST( '20180701 00:00:00' AS datetime) AS [c198],CAST( '20180701 00:00:00' AS datetime) AS [c165] ) ) AS [semijoin2] on ( ([semijoin2].[c198] = [basetable0].[c198]) AND ([semijoin2].[c165] = [basetable0].[c165]) )
I'm still not understanding why this is happening. Why can't PBI join the two tables instead of trying to pass these dates back and forth???
March 31 - April 2, 2025, in Las Vegas, Nevada. Use code FABINSIDER for a $400 discount!
Check out the February 2025 Power BI update to learn about new features.
User | Count |
---|---|
124 | |
114 | |
73 | |
65 | |
46 |