Dataflow Gen2 Issue

[cassidy]

When trying the new "Dataflow Gen2", an error of "There was a problem refreshing the dataflow. Please try again later. (Request ID: aa7e41ba-d6a9-47df-8d67-e5a3c98146b2)." persists.

To test, I have an exact copy of the same Dataflow but non "Gen2".  This one is running without issue, suggesting that the issue is specific to the Gen2.

The source is a SQL Server through a Gateway.  The Gateway is up to date (May 2023)

[StephenBatich]

I think I figured it out!!!

Disable "Staging".  To do this, right click on the tables that are being imported on the left of the Power Query editor, and uncheck the 'enable staging'.  Once I did that, everthing worked fine!!!

[BEEYE-FLY]

we got exactly the same issue, is there any solution to fix this issue?

[StephenBatich]

Disable "Staging" - that worked for me:

[DanielleR]

Hi, I'm getting the same error.

My source is a SQL on premise database, my destination is a lakehouse and I'm using a Dataflow Gen2 via a gateway (version 3000.178.9 (June 2023)) to transfer data into the lakehouse. The same gateway is being used by Power BI and Data Factory with no issues, which leads me to believe it wouldn't be a firewall or gateway service account issue, as otherwise the problem with connecting would exist there too? 

Can anyone please help or advise? Am I missing something?

TIA.

[v-jingzhang]

The underlying issue is that the data gateway cannot connect to the dataflow staging LakeHouse in order to read the data before copying it to the desired output destination. Reading data from a lakehouse uses TCP over port 1433, which is used to copy the data from the staging lakehouse to the output destination. So the firewall rules on the data gateway server and/or customer's proxy servers need to be updated to allow outbound traffic from the data gateway server to the following

Protocol: TCP
Endpoint: *.datawarehouse.pbidedicated.windows.net
Port: 1433

When you use the same data gateway for Power BI and Data Factory with no issues, it is because their traffics flow to different Endpoints or they use different protocols/ports that are not blocked on the data gateway server.

Best Regards,
Community Support Team _ Jing

[jespinosar]

You saved my day! My firewall was filtering the traffic flow you mention!

[v-jingzhang]

Hi all

Sorry for the late reply. Here is the update of this issue. 

Symptom

• Refresh of a Dataflow Gen 2 with output destination failing via gateway. Gateway refreshes in Dataflows Gen2 writes directly to destination from the Gateway (LakeHouse, DataLake, etc.).
• ​Dataflows through a Gateway may fail to reach DataLake, LakeHouse, or other destination if the gateway is not configured to allow connections directly to that destination.

Solutions and Workarounds

There is not going to be a fix for this, this is by design, please refer to below details to overcome the situation:

• The gateway must be configured to be able to pass through the firewall or proxy to reach destination data source. If the user is using a proxy server, this may require enable-listing URLs to appropriate destinations. i.e. (*-datawarehouse.pbidedicated.windows.net) for LakeHouse, (*.dfs.core.windows.net) for DataLake, etc.
• Users using LakeHouse destinations must be running at least the May 2023 release of the gateway (the connector is not available in gateways prior to this release).
• Public doc updates for this are now live - https://learn.microsoft.com/en-us/data-integration/gateway/service-gateway-proxy#configure-gateway-f... .

Best Regards,
Community Support Team _ Jing
If this post helps, please Accept it as Solution to help other members find it.

[DenisLux]

I also have the same issue but I noticed a few particularities:

1) If I create dataflow Gen2 with source from Sharepoint and destination to Warehouse/Lakehouse -> it works

2) If in the same dataflow I add another source from Azure SQL via gateway -> both queries work for tables update but now fail with wiring to destination (Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: Microsoft SQL: A network-related or instance-specific error occurred while establishing a connection to SQL Server)
3) If now I delete second query and leave initial one (which was working), it doesn't work anymore and gives the same error while trying to write to Warehouse.
It seems that if there is gateway in the source -> dataflow loses connection to Warehouse/Lakehouse (even if they are in the same worspace)

[Kevin_H]

Same issue here; my source is also a database, for which I need a data gateway. My destination is a lakehouse. Is this known bug specific to a dataflow that uses a gateway? Or do all Dataflows Gen2 fail to load data into a lakehouse?

This recurs even with gateway version 3000.178.9 (June 2023).

[mirmoazam]

I'm facing a similar issue while i'm trying to follow the Tutorial on Getting started with MS Fabric.

it's quite strange that the Dataflow even if successfully refreshed wouldn't show up in the pipeline

[v-jingzhang]

Hi @cassidy 

This is a known issue of Dataflow Gen2 in Fabric. Engineers are investigating this issue. There is no solution or workaround at present. And there is no ETA for the fix as of now. Please be patient for the fix. We will update here if there is any news. 

Best Regards,
Community Support Team _ Jing
If this post helps, please Accept it as Solution to help other members find it.

[ankushbhatia7]

Is there a fix yet? I am still getting error loading to Lakehouse from On-prem db using dataflow gen 2. 

[StephenBatich]

Disable Staging - fixed it for me:

[cassidy]

I think I got it a little further down the line, but now get this repeat issue:

MashupException.Error: We don't support creating directories in Azure Storage unless they are empty

There is such little info out there for troubleshooting this early on, any advice is appreciated.

[DarinSpence]

I got the Dataflow Gen2 to start working.  Basically, I did a reboot.

1)  Export Dataflow Gen2 to a Power Query Template (.pqt) file

2) Downgrade to a Pro Workspace

3) Upgrade to a Fabric Trial
4) Create Lakehouse

5) Create Dataflow Gen2 using the Power Query Template (.pqt) file.

[BEEYE-FLY]

unfortunately this workaround does not work for our case.

[StephenBatich]

Try disabling the 'staging tables'  -  that resolved the issue for me: