Forum Discussion
Dataflow Gen2 fails when trying to create new table in Data Warehouse
Could you please raise a support ticket and share the ticket number with us? Below is the url to the support page so you can create the ticket:
https://support.fabric.microsoft.com/en-US/support/
im unable to repro this. Are you able to replicate this error / behavior in other workspaces with a different warehouse ?
pqian_MSFT thank you for the information 😀
Do all the dataflows gen2 in a workspace, which have data warehouses as their destination, use the same staging lakehouse?
Do all the dataflows gen2 in a workspace, which are writing to lakehouses, use another staging lakehouse? I have some lakehouses also in this workspace, and those dataflows gen2 which are writing to the lakehouses are not getting this error. They also have staging enabled. So it seems not everything in this workspace is broken...
Obviously, I wish this error didn't happen, because now I need to move everything to another workspace...
But at least, now I know that this is what I need to do 😊
This error is pretty specific to the path that loads data into DW as output destination. In order to efficiently load to DW, the data must be staged in the LH and then bulk transferred to DW via the LH's SQL endpoint.
The "staging lakehouse" is a hidden artifact that is bound to a workspace. It gets created when the first dataflow gen2 gets created in that workspace. We just became aware that LH can get created but the SQL endpoint can fail to provision, leaving us with a semi-functional LH. This is a bug that has to be resolved by us. Unfortunately because the staging LH is hidden from the user, there isn't an easy way to recreate it (you can, for instance, access the staging LH directly if you know its endpoint\ID). The easiest mitigation right now is ditch the workspace.
We are still analyzing why LHs are getting created without functional SQL endpoints - but even if we fix that issue, it likely will not mitigate yours, since it's already broken.