Forum Discussion

markjdm's avatar
markjdm
Regular Visitor
2 years ago

Dataflow Gen2 Failing where Gen1 Works

Hi,   I'm posting here to reach out to anyone who has had issues with Gen2 vs Gen1 dataflows.  I have a table that is about 30m rows a majority of this is line level data which is being pivoted to ...
data engineering
frithjof_v's avatar
frithjof_v
Community Champion
2 years ago

I read some other posts in this forum which also says that they experience the dataflow gen2 to be slower than the Power BI dataflow.

I haven't tried to compare myself.

 

About getting data from a Power BI dataflow into a dataflow gen2 (and then write to a Lakehouse): isn't it possible to use the Dataflows connector to get data from a Power BI dataflow into the Dataflow Gen2? https://learn.microsoft.com/en-us/fabric/data-factory/dataflow-support#d

 

When writing to a Lakehouse destination, it is generally recommended to disable staging.

  • markjdm's avatar
    markjdm
    Regular Visitor
    2 years ago

    Thanks frithjo_v. I can use connectors yes on both gen1 and gen2. Issue is when i run the same query on gen2 it fails. It's been with MS support for more than 1 month now. 

    • frithjof_v's avatar
      frithjof_v
      Community Champion
      2 years ago

      Do you get an error message? What does the error message say?

       

      If you can provide any screenshots, it might be helpful in understanding the issue.

       

      If I understand correctly, you have tried two options:

       

      1. Create a new Dataflow Gen2. Copy the original query from Power BI Dataflow and paste it into a Dataflow Gen2.

      It means the new architecture will be like this:

      Source -> Dataflow Gen2.

      Does the data preview in Dataflow Gen2 work? Or also the data preview fails? 

       

      2. Create a Dataflow Gen 2, and use Get Data to connect to the Power BI Dataflow as a data source and import the data from the Power BI Dataflow.

      It means the new architecture will be like this:

      Source -> Power BI Dataflow -> Dataflow Gen2.

      This also fails? Does the data preview in the Dataflow Gen2 also fail?

       

      General note: In the data preview in the Dataflow Gen2. Could you try selecting all columns and then click Keep rows -> Keep errors. To see if you can identify some error cells.