Forum Discussion

MarcusA's avatar
MarcusA
New Member
2 years ago

DFGen2 Unable to create Destination table in Lakehouse

Hi, trying to create a Destination table in a Lakehouse while using DFGen2 to import data. But I am unable to create tables, both when using Data Destination - Choose destination target dialog, or w...
Dataflows
Element115's avatar
Element115
Icon for Memorable Member rankMemorable Member
2 years ago

Screenshot, code and details please.

Yusuf7's avatar
Yusuf7
Frequent Visitor
2 years ago

Thanks for your enquiry. My dataflow Gen2 were working well since I've created them last week. It's only from this morning they started failing to refresh. I detached the destinations and they were working well. But couldn't reconnect with the destination lakehouse again.

 

 

I had followed the workaround by creatng a staging dataflow and then connect my dataflow gen2 taking data from that staging dataflow. It still didn't work.  

  • Element115's avatar
    Element115
    Icon for Memorable Member rankMemorable Member
    2 years ago

    OK well... if nothing has changed when it stopped working, then it's up to Microsoft to step in here. By the way, we can still see the names you blacked out when we display your screenshot in full screen, the black color is slightly transparent  😉

    • Yusuf7's avatar
      Yusuf7
      Frequent Visitor
      2 years ago

      My bad 😞 Thanks for lettig me know though!!

  • pqian_MSFT's avatar
    pqian_MSFT
    Icon for Microsoft Employee rankMicrosoft Employee
    2 years ago

    This again indicates your GW cannot see the LH server (or cannot connect to it for whatever reason).

    Please gather the following things:

    - Session ID (In the options dialog)

    - Gateway logs with "Additional Logging" turned on while the repro is performed.

     

    P.S., we are working on a change that will show more details when this error happens

    • Yusuf7's avatar
      Yusuf7
      Frequent Visitor
      2 years ago

      My session ID of the failed WriteDataToDestination is is d95209b8-dd1a-439d-9eb9-697cab75c5cf

       

      DataflowID is 795512cf-8e8a-4034-8f29-4641664c8bcb

       

      Tried with the workaround mentioned (staging dataflow extracting data from SharePoint & Second dataflow writing that to destinaion lakehouse), didn't work at all.

       

      The same error not only affected my dataflows writing to the lakehouse/Warehouse, but also occurred with my powerbi connection to dataflow Gen2 & Lakehouse tables.

       

       

      I had just migrated significant portion of my ETL pipelines in fabric dataflow gen2 & Lakehouse, it was working perfectly fine for a week until this suddenly happened, ruining my efforts. 

       

      I'm a business analyst with little understanding of backend security protocols & gateways, therefore, it'd be helpful to have a clearer solution.  

      • pqian_MSFT's avatar
        pqian_MSFT
        Icon for Microsoft Employee rankMicrosoft Employee
        2 years ago

        Yusuf7 this seems to be a different issue than what's reported on this thread. First of all, your dataflow does not use a GW, so the error is entirely on the service side and not related to any firewall rules\etc. Secondly, I can see that your dataflow is now refreshing successfully again - can you please confirm this error went away.

         

        FWIW the error that you reported above is related to your destination lakehouse. Suddenly it became unavailable and then came back online a couple days later. I'll engage the LH team to try to understand what happened to it during the outage period.