User Profile
009co
Helper IV
Joined 5 years ago
User Widgets
Contributions
Re: Dataflow Gen 2 dataflow queries with PIVOT long running, never return results
Hey miguel, it appears that Dataflow Gen 2 simply cannot do the pivot transform. My guess is that it is related to lack of resources. But maybe it is just on my trial account? But are you able to use the data I linked below to replicate my steps?Re: Create new KQL database - stuck on loading indefinitely after being created
Thanks for reply GeethaT Yes that is what I am doing. Here are screenshots: Create new KQL Database However instead of showing the detail page for the new KQL Database it gets stuck indefinitely on the Loading page. If I try to go back to the Workspace and try to open the new KQL Database it gets stuck indefinitely on the Loading page. I have not yet been able to create a KQL Database successfully in many many attempts. It always gets stuck on the loading page and never goes past it. Something in my Fabric instance seems to be off.Re: Dataflow Gen 2 dataflow queries with PIVOT long running, never return results
Thanks for following up. Cardinality is low. Parameter column has only 3 distinct values eg pivots to give 3 columns. It completes quickly successfully in Power BI Desktop. I created Dataset containing the 4 Lakehouse Tables, then exported Dataset tables to csv and put them here https://github.com/sitrucp/Deadman-River Test: * Upload csv files to Lakehouse Tables * Create Dataflow Gen2 query that => appends 4 Tables => Pivot appended data on Parameter column with Value resulting in one row per Date with 3 columns (Discharge, Level, Precipitation) => Data destination to new Lakehouse Table. => Run Dataflow Gen2 Edit to add: I actually did the test described above in a brand new Workspace, Lakehouse, Dataflow Gen2, etc and got same results eg the dataflow with the pivot will not complete and not ever create new Lakehouse Table. ----------------------------- More details on dataflow publish fail / message that I get: The dataflow could not be saved. The Model evaluation was cancelled. Please try again later. Session ID: 9b7d629f-404f-0562-9575-1095d2ed09ee Root activity ID:: 31686fac-a385-4266-bdb8-054688650a55 Time: 2023-06-05T13:59:40.677ZRe: Lakehouse staging "ghosts" - intended to be hidden but show when dataflow gen 2 dataflows created
From Dataflow Gen 2 Known Issues: https://learn.microsoft.com/en-us/fabric/data-factory/known-issue-staging-artifact Known issue - Staging artifacts aren't hidden and can be modified by the end-user Article 06/06/2023 2 contributors Feedback In this article Symptoms Solutions and workarounds In a new workspace, when an end-user first launches a Dataflow Gen2, a set of staging artifacts, such as a new Lakehouse and a new Data Warehouse, are automatically created for the whole workspace. These artifacts can be used for a set of functionalities available for Dataflow Gen2. A user is able to see these staging artifacts in the workspace list and lineage and being able to interact with these artifacts. If a user deletes or modifies any of these staging artifacts, the workspace can be set in a state that won't be able to use these artifacts because of misconfiguration. APPLIES TO: ✔️ Dataflow Gen2 in Microsoft Fabric Status: Open Problem area: Data Factory Symptoms When trying to refresh a Dataflow gen2 and the error refers to: null Error: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: PowerBIEntityNotFound: PowerBIEntityNotFound After deleting the staging artifacts, you're unable to get them back. Solutions and workarounds Recreate your Dataflow Gen2 solution in a new workspace.Re: Dataflow Gen 2 dataflow queries with PIVOT long running, never return results
Hi thanks, here is the query m-code. * If Pivot step is removed the query completes successfully, otherwise it doesn't has described in this thread. * The combined / appended tables have about 4500 records. * Parameter column has 3 distinct values eg pivots to give 3 columns.. let Source = Table.Combine({#"2011_to_2021_level_and_discharge", #"2013_to_2023_precipitation", #"2021_to_present_discharge", #"2021_to_present_level"}), #"Changed column type" = Table.TransformColumnTypes(Source, {{"Parameter", type text}, {"Value", type number}, {"Date", type datetime}}), #"Pivoted column" = Table.Pivot(Table.TransformColumnTypes(#"Changed column type", {{"Parameter", type text}}), List.Distinct(Table.TransformColumnTypes(#"Changed column type", {{"Parameter", type text}})[Parameter]), "Parameter", "Value") in #"Pivoted column"Re: Dataflow Gen 2 dataflow queries with PIVOT long running, never return results
When you mention that this has never returned results, you mean during a refresh? or while you were inside the Power Query editor? Both. Below is the dataflow query (named Data), that appends 4 tables and pivots on string column, waiting to show results. Note that the Source step does quickly show the appended results. It is the Pivot step that is the issue. Below is the dataflow (named Dataflow 2) which is the above but Published, waiting to show results. Below is screenshot of the above screen after maybe 20 or 30 minutes when it stops, shows the highlighted red triangle beside dataflow name, that pops the message about "The dataflow was could not be saved. The Model evaluation was cancelled. Please try again later." Below is query plan which is also waiting to show results (I guess the query has to return results before the query plan can be shown?)
