Dataflow Gen2 Failing where Gen1 Works

[markjdm]

Hi,

I'm posting here to reach out to anyone who has had issues with Gen2 vs Gen1 dataflows.  I have a table that is about 30m rows a majority of this is line level data which is being pivoted to flatten the table. The pivot will be doing about 20 new columns by filtering out rows that aren't needed.   The dilemma here is that a Gen2 dataflow will always 100% fail witht exact same query that is run on a gen 1 dataflow.  Note: Both gen1 and gen2 dataflows would be configured to "enable staging" and the Gen2 is not loading into a lakhouse/warehouse for testing purposes.  

On another note, trying to get data from a gen1 dataflow into a gen2dataflow then into a lakhouse doesn't seem to work either? Anyone else have this experience? 

Cheers,

Mark

[Anonymous]

Hi @markjdm ,

Dataflow Gen2 is the new generation of dataflow. You can read the following article to understand the difference between Dataflow Gen1 and Dataflow Gen2:
Differences between Dataflow Gen1 and Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

If you are experiencing the exact same query failure in a Gen2 dataflow, consider the following steps to troubleshoot the problem:

1. Ensure that the Gen2 data stream has the necessary fabric capacity.
2. Verify that the query does not exceed Gen2's functional or resource limits.

If the suggestions above do not help, please provide a specific error message so that we can identify the issue for you.

For moving data from Gen1 to Gen2 and then to Lake Warehouse All-in-One, please see the following links:

Move queries from Dataflow Gen1 to Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

Copy sample data into Lakehouse and transform with dataflow - Microsoft Fabric | Microsoft Learn

Best Regards,

Ada Wang

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[markjdm]

Hi Ada,

Thanks I've already done most of those. I can't see any document that has the key limitations of gen2. I'm assuming it should have similar or better limitations vs gen 1. However the primary issue is gen1 refreshes sucessfully 100% of the time will gen2 fails 100% of the time.  There shouldn't be this kind of discprepancy on a supposed gen2? 

On another thing to note on the same dataflow.  I tried using the gen1 dataflow and used that as a source for a gen2 dataflow. Even that fails? 

Cheers,

Mark

[frithjof_v]

I read some other posts in this forum which also says that they experience the dataflow gen2 to be slower than the Power BI dataflow.

I haven't tried to compare myself.

About getting data from a Power BI dataflow into a dataflow gen2 (and then write to a Lakehouse): isn't it possible to use the Dataflows connector to get data from a Power BI dataflow into the Dataflow Gen2? https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-support#d

When writing to a Lakehouse destination, it is generally recommended to disable staging.

[markjdm]

Thanks frithjo_v. I can use connectors yes on both gen1 and gen2. Issue is when i run the same query on gen2 it fails. It's been with MS support for more than 1 month now. 

[frithjof_v]

Do you get an error message? What does the error message say?

If you can provide any screenshots, it might be helpful in understanding the issue.

If I understand correctly, you have tried two options:

1. Create a new Dataflow Gen2. Copy the original query from Power BI Dataflow and paste it into a Dataflow Gen2.

It means the new architecture will be like this:

Source -> Dataflow Gen2.

Does the data preview in Dataflow Gen2 work? Or also the data preview fails? 

2. Create a Dataflow Gen 2, and use Get Data to connect to the Power BI Dataflow as a data source and import the data from the Power BI Dataflow.

It means the new architecture will be like this:

Source -> Power BI Dataflow -> Dataflow Gen2.

This also fails? Does the data preview in the Dataflow Gen2 also fail?

General note: In the data preview in the Dataflow Gen2. Could you try selecting all columns and then click Keep rows -> Keep errors. To see if you can identify some error cells.