topic "Enable Staging" Behaviour with Create Table option in Dataflow

"Enable Staging" Behaviour with Create Table option

MOVC — Tue, 31 Oct 2023 16:31:36 GMT

Is there some documentation regarding how exactly the Enable Staging option works within Dataflow Gen2? We are having issues where is staging isn't enabled then we cannot use the Create Table options in Lakehouse and Warehouse and instead need to create them ahead of time which isn't a big deal but is tedious. I don't know if this is intentional or not. Especially since we're experiencing the long loading time other people are having when it is enabled.

Re: "Enable Staging" Behaviour with Create Table option

MOVC — Tue, 31 Oct 2023 16:39:43 GMT

Sorry I should have said, we can create Gen2 Dataflows with the Create Table option enabled but if Enable Staging is disable then the table doesn't actually get created, for both Lakehouses and Warehouses.

Re: "Enable Staging" Behaviour with Create Table option

miquelladeboer — Tue, 31 Oct 2023 16:58:48 GMT

You should be able to create new tables to the Warehouse or Lakehouse regardless if staging is enabled or not.

I am not able to reproduce your issue. Can you please open a support ticket, so we can better investigate?

Re: "Enable Staging" Behaviour with Create Table option

miquelladeboer — Wed, 01 Nov 2023 15:50:46 GMT

There was a regression that caused this behavior. A solution has been deployed and should be available in all regions by the end of this week.

Re: "Enable Staging" Behaviour with Create Table option

Anonymous — Fri, 03 Nov 2023 10:57:51 GMT

Hi MOVC ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. Please let me know if you have any further queries.

Re: "Enable Staging" Behaviour with Create Table option

MOVC — Fri, 03 Nov 2023 11:11:51 GMT

I have tested this morning and was able to successfully create a new Warehouse table with staging disabled through a gen2 Dataflow so this is now resolved.