Gen2 Dataflows suddenly failing

motoray Thanks to some guidance from MS engineers, we finally stumbled on the solution and it is : DO NOT FOLLOW the instructions from Microsoft documentation here: On-premises data gateway considerations for data destinations in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn

My sysadmin had added all 3 endpoints to the FW rules and it didn't work. In fact, deleting them and their respective rules, and replacing them with one rule for the MS-SQL Server service TCP/1433 fixed everything!  

DATAFLOW and ON-PREM DB connectivity solution - Microsoft Fabric Community

Been a while now I had our sysadmin update the firewall rules and multiple DFs had no issue ingesting from the on-prem DB until last Friday.   Although last Friday, I asked the sysadmin to add this to the FW rules, as per the MS documentation, as this endpoint is new and wasn't mentioned a couple of weeks ago:  *.dfs.fabric.microsoft.com  .

Also, should have said this was another test of DF at the behest of the SE, case #2401190040000098.  

I notified the SE of the failure of the test he asked me to perform, so I guess, no need to open a new support ticket.  This is what he asked me to do:

We got an update from Product team.
We suspect the issue could be related to missing configuration in the Gateway to allow reaching to the Lakehouse storage.

Please try out the following steps and provide the outcome of two refreshes:
1. Create a new dataflow A with a simple query (blank table) that is using the Lakehouse storage as output destination and then try to refresh the dataflow.
2. Create a new Dataflow B with an entity that uses a data source that utilizes the GW and configure the Lakehouse as the output destination and then refresh the Dataflow.