BUG::Data destination can't be set. Vague error msg produced.

[Element115]

In a new data flow, when I try to set the data destination to an existing lakehouse as a new table, this error message shows up.  I tried later again and again, and now it's been 4+ hours this is going on.  What's happening with Fabric? Something down?

[v-cboorla-msft · Accepted solution]

Hi @Element115 

Thanks for using Microsoft Fabric Community.

Apologies for the inconvenience.

I tried to repro the scenario and didn't find any difficulty in choosing the data destination as Lakehouse. For reference please find the screenshot.

If you are using an on-premises as data source with Dataflow Gateway might cause connectivity problems. The firewall might be blocking communication between the gateway and the destination (lakehouse or warehouse).

For more details please refer : Solution: Set new firewall rules on server running the gateway

In case the issue still persists, please do let us know. Glad to help.

I hope this information helps.

Thanks.

[Element115]

Thanks to some guidance from MS engineers, we finally stumbled on the solution and it is : DO NOT FOLLOW the instructions from Microsoft documentation here: On-premises data gateway considerations for data destinations in Dataflow Gen2 - Microsoft Fabric | ...

My sysadmin had added all 3 endpoints to the FW rules and it didn't work. In fact, deleting them and their respective rules, and replacing them with one rule for the MS-SQL Server service TCP/1433 fixed everything!  

DATAFLOW and ON-PREM DB connectivity solution - Microsoft Fabric Community

[Element115]

Goddamn!   I clicked the 'Accept as solution' by mistake.'

Just wanted to add to my previous reply:

Been a while now I had our sysadmin update the firewall rules and multiple DFs had no issue ingesting from the on-prem DB until last Friday.   Although last Friday, I asked the sysadmin to add this to the FW rules, as per the MS documentation, as this endpoint is new and wasn't mentioned a couple of weeks ago:  *.dfs.fabric.microsoft.com  , and it went downhill from there.

Also, should have said this was another test of DF at the behest of the SE, case #2401190040000098.  

I notified the SE of the failure of the test he asked me to perform, so I guess, no need to open a new support ticket.  This is what he asked me to do:

We got an update from Product team.
We suspect the issue could be related to missing configuration in the Gateway to allow reaching to the Lakehouse storage.

Please try out the following steps and provide the outcome of two refreshes:
1. Create a new dataflow A with a simple query (blank table) that is using the Lakehouse storage as output destination and then try to refresh the dataflow.
2. Create a new Dataflow B with an entity that uses a data source that utilizes the GW and configure the Lakehouse as the output destination and then refresh the Dataflow.

[Element115]

Re the firewall, already done a long time ago. Also already tested a couple of DFs ingesting from the on-prem DB and this did not occur at the time.  

Also, should have said this was another test at the behest of the SE, case #2401190040000098.