Gen2 Dataflows suddenly failing

[motoray]

Hello,

We have several Gen2 dataflows that grab data from SharePoint and send it to a data warehouse. These have been working consistently for at least 2 months. They failed yesterday and I can't get them to work anymore without rebuilding them. Then they work one time and won't refresh anymore after that.

Here's my error message:

There was a problem refreshing the dataflow. Please try again later. (Request ID: 4c48163b-f99b-4252-9928-8395e4d24de1).

Googling made it sound like it's a problem with the firewall blocking port 1433 on my end, but nothing has changed WRT our firewall. Also, I don't undertand why it would work once and then fail on refresh. 

As part of the troubleshooting, we updated to the latest version of the gateway (as of 3/1/2024), but this did not help.

Any help would be appreciated.

Thanks,

Ray

[miguel]

Hi!

id recommend reaching out to the support team by raising a support ticket so our team of engineers can take a closer look at your specific Dataflows:

https://support.fabric.microsoft.com/support

[Element115]

@motoray Thanks to some guidance from MS engineers, we finally stumbled on the solution and it is : DO NOT FOLLOW the instructions from Microsoft documentation here: On-premises data gateway considerations for data destinations in Dataflow Gen2 - Microsoft Fabric | ...

My sysadmin had added all 3 endpoints to the FW rules and it didn't work. In fact, deleting them and their respective rules, and replacing them with one rule for the MS-SQL Server service TCP/1433 fixed everything!  

DATAFLOW and ON-PREM DB connectivity solution - Microsoft Fabric Community

[Element115]

Been a while now I had our sysadmin update the firewall rules and multiple DFs had no issue ingesting from the on-prem DB until last Friday.   Although last Friday, I asked the sysadmin to add this to the FW rules, as per the MS documentation, as this endpoint is new and wasn't mentioned a couple of weeks ago:  *.dfs.fabric.microsoft.com  .

Also, should have said this was another test of DF at the behest of the SE, case #2401190040000098.  

I notified the SE of the failure of the test he asked me to perform, so I guess, no need to open a new support ticket.  This is what he asked me to do:

We got an update from Product team.
We suspect the issue could be related to missing configuration in the Gateway to allow reaching to the Lakehouse storage.

Please try out the following steps and provide the outcome of two refreshes:
1. Create a new dataflow A with a simple query (blank table) that is using the Lakehouse storage as output destination and then try to refresh the dataflow.
2. Create a new Dataflow B with an entity that uses a data source that utilizes the GW and configure the Lakehouse as the output destination and then refresh the Dataflow.

[Element115]

Something is going on.  I got this error message for the past 3 hours now when trying to specify on of my lakehouse as a data destination from within a new dataflow. Everything ground to a halt, no work's getting done.  Later has been 3 hours going on 4. What is left to say at this point?

[motoray · Accepted solution]

Magically things started working again.

Because magic.

[Element115]

LMHO YMMD 🤣