Forum Discussion
Helper IVDataflow Gen 2 dataflow queries with PIVOT long running, never return results
Community Admincould you please share your dataset with us so we could try and repro this scenario?
The only thing that I can think of is that your Parameter column might have a high cardinality where it tries to create over 250 new columns using the Pivot operation, but its something that we would love to learn from and improve the experience.
Helper IVThanks for following up.
Cardinality is low. Parameter column has only 3 distinct values eg pivots to give 3 columns. It completes quickly successfully in Power BI Desktop.
I created Dataset containing the 4 Lakehouse Tables, then exported Dataset tables to csv and put them here https://github.com/sitrucp/Deadman-River
Test:
* Upload csv files to Lakehouse Tables
* Create Dataflow Gen2 query that
=> appends 4 Tables
=> Pivot appended data on Parameter column with Value resulting in one row per Date with 3 columns (Discharge, Level, Precipitation)
=> Data destination to new Lakehouse Table.
=> Run Dataflow Gen2
Edit to add:
I actually did the test described above in a brand new Workspace, Lakehouse, Dataflow Gen2, etc and got same results eg the dataflow with the pivot will not complete and not ever create new Lakehouse Table.
-----------------------------
More details on dataflow publish fail / message that I get:
The dataflow could not be saved. The Model evaluation was cancelled.
Session ID: 9b7d629f-404f-0562-9575-1095d2ed09ee
Root activity ID:: 31686fac-a385-4266-bdb8-054688650a55
Time: 2023-06-05T13:59:40.677Z