general question
13 TopicsHigh volume Dataverse dataflow times out avoid NESTED FROMs and/or JOINs best approach to resolve?
Background We have a Gen1 Dataverse dataflow that calculates aggregate totals per record (yearly, prior-year, lifetime, and household rollups) and writes them to two output entities, Merge Account and Merge Contact. Each output entity is built from a chain of ~11 Table.NestedJoin (JoinKind.FullOuter) joins over ~12 upstream aggregate queries. The problem The dataflow refreshes without issue on smaller environments but times out on high-volume ones with: Microsoft SQL: "The updated two minutes (2) timeout period elapsed prior to completion of the operation. Please avoid using SELECT *, or NESTED FROMs and/or JOINs to execute the query with default five minutes (5) timeout." ErrorCode = 10478; Number = 40000 (Some environments show error 40197 instead.) It stops on the Merge step. What we've implemented so far We've built a monitoring solution: a cloud flow that detects dataflow refresh failures and logs the details (dataflow name, ID, and error message) into a custom Refresh Audit table, surfaced in a model-driven app so the team has visibility into failures. We've analyzed the failure across multiple environments and confirmed it correlates with data volume — the same dataflow completes on smaller datasets and times out on larger ones, with the filtered row volume and per-record density appearing to be the main drivers. We've identified that the full-outer joins don't appear to fold, so a large nested query is sent to the Dataverse TDS endpoint and hits the query timeout — i.e. it's a timeout, not a data/row error. Approaches we're evaluating — would these be effective? Increasing the query/command timeout for a Gen1 Dataverse dataflow — is that configurable, and where? Using Table.Buffer on the join inputs to force local evaluation and avoid pushing the nested query to the TDS endpoint — would this reliably help, or just shift the bottleneck? Staging the upstream queries into computed/staged entities (or a separate staging dataflow) — is this the recommended pattern, and does it need enhanced compute / premium? For very large datasets (1M+ rows), pushing the aggregation source-side (FetchXML aggregate / SQL view) — is this the more reliable long-term approach? Is there a supported way to get per-step timing / query diagnostics for a dataflow refresh, to pinpoint the heaviest step and enrich our monitoring? Any guidance on which of these works best for high-volume Dataverse dataflows with heavy full-outer join chains would be much appreciated. Thanks!Solved341Views0likes5CommentsGrouping Values by Multiple Criteria in several Tables (DAX or Power Query and big Flat Table)
I have don’t this in the past with data from our ERP with Power Query when it was one big flat table. However, I am hitting a big wall trying to do this in Power BI with many related tables. I need to create some groupings in Power BI for tables and charts. I have looked at several examples, but most seem to have a less complicated data model. I had considered Power Query, but I need to kook at multiple related tables and did not want a massive flat table again. The DAX I saw was confusing and there seemed to be some cautions. I have 5-7 different groupings (A, B, C, D, E…). I have actual expense and budget tables. In order for an amount to be summed into one of the categories it must meet several criteria that are in different related tables. It must Be in a certain Account Rage Be in certain Cost Centers, Be a specific Project Type Output might look like this Category 2025 Actuals YTD 2025 Projection 2026 Budget 2027 Budget 2028 Budget A 14143 14134 326 3453 433 B 2356 354 3533 5353 3452 C 3423 3497 3534 4345 3455 D 2324 2345 3533 3535 5352 E 2567 2599 5632 3567 3572 I can create a single measure to do one category but, I am really after either a “real” table via Power Query or a DAX query. Below is an example a measure if I were creating category A as a single measure for Actuals. Help ang guidance from multiple angles would be appreciated. Thanks Alan For example, for Category A measure might look like this, unfortunately I need that as a “row” not a new column. A = VAR _Amount = CALCULATE ( SUM ( 'Actuals'[Amount4] ), KEEPFILTERS ( VALUE ( 'Account Relations'[Head Account] ) >= 50000 && VALUE ( 'Account Relations'[Head Account] ) <= 69999 && VALUE ( 'Account Relations'[Head Account] ) <> 56950 ), KEEPFILTERS ( VALUE ( 'Account Relations'[Account] ) <> 96910 && VALUE ( 'Account Relations'[Account] ) <> 96950 ), KEEPFILTERS ( VALUE ( 'Main Project Master File - PMF'[PMF Project Type] ) >= 30 && VALUE ( 'Main Project Master File - PMF'[PMF Project Type] ) <= 69 ), KEEPFILTERS( VALUE('Department Relations'[Cost Center]) >=10 && VALUE( 'Department Relations'[Cost Center]) <= 20 ) ) RETURN _AmountSolved1.2KViews1like5CommentsNeed to create DAX for last 3 month visible in clustered column chart
Hi Experts, I want to create a dax to display the last 3 months when we select any month from the slicer. if not selected then it will display all month data in clustered column chart. Monthname is a slicer (MonthName = FORMAT([Date],"MMMM")) //text type Monthyear is a X axis in visual (MonthYear = FORMAT([Date], "MMM-yyyy")) //text type Tried this measure but no luck: Last3Months = VAR HasMonthSelected = ISFILTERED ( 'CustomCalendar'[MonthName] ) VAR EndDate = CALCULATE ( MAX ( 'CustomCalendar'[Date] ), ALLSELECTED ( 'CustomCalendar' ) ) VAR SelectedDate = MAX ( 'CustomCalendar'[Date] ) VAR Result = SWITCH ( TRUE (), HasMonthSelected && SelectedDate IN DATESINPERIOD ( 'CustomCalendar'[Date], EndDate, -3, MONTH ), 1, HasMonthSelected, 0, 1 ) RETURN Result Please help me on this. Thanks DKSolved1.4KViews0likes8CommentsReport builder - 3 vars in a graph with different timestamp
Hi, I have the data you see in Table, always 3 variables with different timestamp. I just want to plot 3 vars on one grafic, but it seems no to interpolate or something, How can I configure de graph? can you help me? Thanks in advanceSolved872Views2likes4Commentsdataflow
In the below scenario assume that D is the table I use in the report which is a append of 1, A and B. If I refresh DF1 and DF A then its obvious that the last dataflow would also refresh since it has 1 and A as linked entities. Questions: 1. Does the final dataflow refresh twice? 2. If I refresh 1 and A together will it cause the last one, and hence the entire chain to fail? Any best practices? 3. If I care most about A and expect 1 to fail often, how can I ensure that the final DF does not fail? And my append happens with A and B with the new data and 1 with the old data (in the current scenario the entire flow would fail)Solved1.3KViews0likes4CommentsHow to pass cell value as the Query value for a query utilizing Power Query
Good Morning, I am very green to Power Query, so I'm unsure if this is even possible. Base Goal: Pass the value from "SELECT_COL" as the UserQuery "let" statement. Clarifiying Details: I have 2 datasets 1) SURVEY_PARAMS 2) SOURCE_LKP_FILE I have successfully been able to filter down the SOURCE_LKP_FILE until I get just the 1 cell value "SELECT_COL" Code to get to just the query value I want to pass into the UserQuery(I can modify my SOURCE_LKP_FILE and add in or remove anything that's needed to help make this dynamic) below: let Source = SOURCE_LKP_FILE, #"Filtered rows" = Table.SelectRows(Source, each [TABLE_NAME] = "FOWLER_LKP_TBL"), #"Choose columns" = Table.SelectColumns(#"Filtered rows", {"SELECT_COL"}), #"Drill down" = #"Choose columns"{0}[SELECT_COL] in #"Drill down" I want to take the cell value from above and use it as part/whole of my UserQuery(my user query will just be a series of extract by range) I'm a heavy pipeline adf worker and I thought I'd be able to do some form of referencing within here like: let Source = SURVEY_PARAMS, #"Inserted text range" = $SELECT_COL in #"Inserted text range" Any help would be GREATLY appreciated811Views0likes3CommentsParquet File Loading in Dataflow
Hi I tried loading parquet file from my OneDrive of size 200MB in power bi dataflow but it throw an error of Parquet.Document cannot be used with streamed binary values. It works well with parquet file of smaller size. I am attaching the screenshot with the error. is there any solution for this error?Solved3.4KViews1like6CommentsPower BI Desktop Version question
Hi, I understand that there is an optimized version of Power BI Desktop for the January 2024 release of Report Server as specified here. However my question is would the December 2023 version of Power BI Desktop still be able to publish understanding that not all features are supported?Solved1KViews0likes2CommentsIs there a way to display a category count as "less than #" if the count is less than a value?
Hello wonderful BI-ers, I am working with FERPA protected data and cannot display the count of a demographic category if the count is very small, because the person/people in that category might be able to be identified. So instead, I want to display the count of a category as "less than 5" if the count is less than five, but retain the numeric count values for categories where the count is over five. This count would be in a tooltip. In other words: when I hover over a column in my bar chart, I want to display a text value if the count is under a threshold, and a numeric value if the count is over a threshold. Here is a sample of the data (not actual numbers): Year Race Count 2020 White 189 2020 Black 27 2020 Asian 89 2020 Other 2 2019 White 234 2019 Black 54 2019 Asian 4 2019 Other 6 So in this case, when I hover over the bar for [Year=2020 Race=Other], and [Year=2019 Race=Asian], Instead of the tooltip displaying "2" and "4" respecively, I see "Less than 5" (or "< 5"). I have a tooltip set up right now displaying counts in categories, what I'm asking is about displaying those values as text based on a condition. Is this possible?Solved1.6KViews0likes2Comments