Forum Discussion
Memorable MemberBUG::New table in data warehouse not detected
Microsoft EmployeeThe DataflowStagingLakehouse/DataflowStagingWarehouse are reserved artifacts used by the dataflow engine internally to stage data and should not be used as output destinations (we are working on improving the UI to prevent this scenario).
The way I see it, you should be able to create a single dataflow that pulls your data from the on-prem SQL server and pushes the data into a WH you created in the same workspace where your dataflow is running. In this case, the source query, the one that pulls the data from the on-prem SQL server and that has the destination set up, can be configured with staging enabled or disabled (and the Publish button should be enabled since you have at least an output destination). Depending on whether you are performing transformation on top of the data being moved, you can expect perf gains by enabling staging (but you would need to structure your query effectively). Chances are that in your scenario staging doesn't have any positive impact - and it could actually make sense to disable it.
I'm following up internally on the 5hours long running dataflow. It seems to be a known issue and we'll work on a fix.
Thanks for continuing to report feedback!
Memorable MemberBy the way, I am trying to repurpose a third dataflow to chain it to a fourth one, so to do the same thing I did with DF1 and DF2 but against my own created lakehouse instead of the staging lakehouse and I come across multiple issues... one of which is the following:
Since DF3 already had a destination set, I delete it, publish, and then re-open DF3 only to see that the destination is set again EVEN THOUGH I deleted it! So to be clear, after choosing Delete in the menu you see in the screenshot, and Publishing the DF, the delete did not have any effect even though the UI showed that the destination got cleared.