Forum Discussion
DataFlow Gen2 Issue
Hi,
Could someone please help me?
I'm trying to load data from my on-premises SQL Server database into a Lakehouse in Microsoft Fabric.
Step 1: Load data into Power Query (Dataflow Gen2) from SQL Server on-premises.
Result: Success.
Step 2: Load the data into my Lakehouse. This is where I'm encountering the problem.
When I try to connect to the Lakehouse destination, I receive the following error:
Additional information:
- The SQL Server connection is working correctly.
- The On-premises Data Gateway status is Online.
- I can access and write to the Lakehouse successfully from Fabric Notebooks.
- The issue occurs only when trying to use the Lakehouse as the destination in Dataflow Gen2.
Has anyone experienced a similar issue or found a solution?
Thank you.
6 Replies
- LUISS252New Member
Some changes were applied at the firewall level, but honestly, I don't think that was the root cause of the issue. When checking the error on the gateway, it showed a set of successful tests that included validating communication to the address specified in the original error:
'Please check your network connectivity to ensure your on-premises data gateway can access scu.frontend.clouddatahub.net and restart the service to retry the connection again once the connectivity issue has been resolved.'
In addition to this problem, I was also having issues using the Copy Data activity in the pipelines; in that case, it couldn't even reach the server. Interestingly, after some maintenance on the Azure firewall (my 'on-premises' server is actually hosted in Azure), the pipeline Copy Data activity started working.
However, as of today, the original error remains unresolved.Regarding Dataflow Gen2, I have ruled out server-side and gateway errors since I am able to successfully resolve the SQL query on the server side. The issue seems to be related to tenant or workspace configurations, as the error triggers specifically when trying to configure the Lakehouse as the data destination.
At this point, I have tried most of your recommendations, and the one that makes the most sense to me is the one from Prince0011. However, I have validated the workspace identity, and it is configured correctly. That being said, I don't have the 'Azure connections' setting configured in the workspace. Since I don't have tenant-level privileges, I will try to get the administrator's assistance with this and will let you know how it goes.
- Prince0011Solution Sage
Hi,
That scu.frontend.clouddatahub.net check is specifically the Fabric backend endpoint used for Dataflow Gen2 to Lakehouse writes, separate from your regular SQL/Gateway path, so it makes sense Copy Data started working after firewall maintenance while this one didn't, they're hitting different endpoints entirely.
Good catch on Azure connections, that setting is often the missing piece even when Workspace Identity itself looks fine. Let us know what the admin finds.
- v-saisrao-msftCommunity Support
HI LUISS252,
Since the SQL Server source is working, reconfigure the Lakehouse destination, reauthenticate, and publish the Dataflow again, as recommended for destination validation exceptions. If the issue persists, check the destination availability, permissions, and recent changes. If a proxy is used with the gateway, review its configuration and ensure the gateway is up to date. If queries reference other queries or staged data, also verify outbound TCP 1433 connectivity from the gateway/proxy to the required Fabric endpoints.
On-premises and virtual network data gateway considerations for data destinations in Dataflow Gen2 …
Using an On-Premises Data Gateway in Power Platform Dataflows - Power Query | Microsoft Learn
Dataflow Gen2 data destinations validation rules - Microsoft Fabric | Microsoft Learn
Thank you.
- Prince0011Solution Sage
Hi,
Since Notebook writes work fine but Dataflow Gen2's Lakehouse destination doesn't, this usually comes down to workspace identity or staging permissions, not the Gateway itself.
Check that your workspace has a Workspace Identity enabled, Dataflow Gen2 destinations to Lakehouse rely on it for the write step, Notebooks don't need this the same way.
Also confirm the Dataflow's staging settings, if "Fast copy" or staging is enabled and misconfigured, Lakehouse destination writes can silently fail even when the source connection itself is fine.
If this helped, feel free to give it a kudos or mark it as solution, helps others find it too.
If your problem can't solve, please let me know
- Murtaza_GhafoorSuper User
I wouldn't start by changing the Lakehouse permissions or recreating the Lakehouse.
The message doesn't indicate an authorization failure such as 401/403; instead, the connection itself is being closed unexpectedly.
Please check On Premises Gateway connection, since your original sources is on prem SQL, so you need to make connection for that so you can dump the data in lakehouse.
2nd option, you might need to check is completely check firewall settings, some ports or internet traffic may be blocked. which is causing the connection to break.
If this helps, ✓ Mark as Kudos | Help Other