Forum Discussion
DFGen2 Unable to create Destination table in Lakehouse
Thanks for your enquiry. My dataflow Gen2 were working well since I've created them last week. It's only from this morning they started failing to refresh. I detached the destinations and they were working well. But couldn't reconnect with the destination lakehouse again.
I had followed the workaround by creatng a staging dataflow and then connect my dataflow gen2 taking data from that staging dataflow. It still didn't work.
This again indicates your GW cannot see the LH server (or cannot connect to it for whatever reason).
Please gather the following things:
- Session ID (In the options dialog)
- Gateway logs with "Additional Logging" turned on while the repro is performed.
P.S., we are working on a change that will show more details when this error happens
Hi
Thanks for your response. Yes the WriteDataToDestination problem for the dataflow along with all the mentioned problems had been automatically resolved and back to normal after two days from when they’ve started (Friday 19th April) all of a sudden.
Our customer reportings and ETL processes are getting migrated to fabric warehouse and lakehouses through data factory experience, and these kind of outages potentially happening again makes myself a bit worried.
I had experienced the same problem mentioned in the thread (screenshot attached, couldn’t select a destination). Besides, all Fabric items were inaccessible from powerbi desktop. Dataflow gen2 items were inaccessible and could write to the destination lake house/warehouses.I had used the workaround mentioned in this Microsoft article, but it didn’t fix anything (using a staging and destination injesting dataflow as mentioned). https://learn.microsoft.com/en-us/fabric/data-factory/gateway-considerations-output-destinations#solution-set-new-firewall-rules-on-server-running-the-gateway
I appreciate your help on this issue and hope that the outage won’t randomly occur again.My session ID of the failed WriteDataToDestination is is d95209b8-dd1a-439d-9eb9-697cab75c5cf
DataflowID is 795512cf-8e8a-4034-8f29-4641664c8bcb
Tried with the workaround mentioned (staging dataflow extracting data from SharePoint & Second dataflow writing that to destinaion lakehouse), didn't work at all.
The same error not only affected my dataflows writing to the lakehouse/Warehouse, but also occurred with my powerbi connection to dataflow Gen2 & Lakehouse tables.
I had just migrated significant portion of my ETL pipelines in fabric dataflow gen2 & Lakehouse, it was working perfectly fine for a week until this suddenly happened, ruining my efforts.
I'm a business analyst with little understanding of backend security protocols & gateways, therefore, it'd be helpful to have a clearer solution.
Yusuf7 this seems to be a different issue than what's reported on this thread. First of all, your dataflow does not use a GW, so the error is entirely on the service side and not related to any firewall rules\etc. Secondly, I can see that your dataflow is now refreshing successfully again - can you please confirm this error went away.
FWIW the error that you reported above is related to your destination lakehouse. Suddenly it became unavailable and then came back online a couple days later. I'll engage the LH team to try to understand what happened to it during the outage period.
Unfortunately this means Fabric is neither ready for real-time or near real-time data processing nor for scenarios that require a 99.999% uptime guarantee. This is bad news especially for businesses in the financial industry that need to keep a live finger (and not a sleepy finger) on the pulse of the market, and other time critical business cases.
I can imagine it is not a trivial challenge for Microsoft to provide such a SLA. Nonetheless, perhaps the marketing department should take its cue from the engineering department, yes, before putting the cart in front of the horse?
