Forum Discussion
Memorable MemberBUG::New table in data warehouse not detected
Memorable MemberRe the refresh ID, it is still refreshing so everything is greyed out for now.
Yes, both DFs belong to the same workspace. I did not disable Staging for DF1 when I started this test. But I did disable Staging for DF2 because I didn't want the data to be stored in the default lakehouse, rather in a lakehouse I created. So I just opened DF1 and disabled Staging now BUT then the Publish button disappears! Dunno if it is because DF2 refresh is still running and depending on DF1, or whether the button would disappear regardless when first creating the DF.
Microsoft EmployeeThe DataflowStagingLakehouse/DataflowStagingWarehouse are reserved artifacts used by the dataflow engine internally to stage data and should not be used as output destinations (we are working on improving the UI to prevent this scenario).
The way I see it, you should be able to create a single dataflow that pulls your data from the on-prem SQL server and pushes the data into a WH you created in the same workspace where your dataflow is running. In this case, the source query, the one that pulls the data from the on-prem SQL server and that has the destination set up, can be configured with staging enabled or disabled (and the Publish button should be enabled since you have at least an output destination). Depending on whether you are performing transformation on top of the data being moved, you can expect perf gains by enabling staging (but you would need to structure your query effectively). Chances are that in your scenario staging doesn't have any positive impact - and it could actually make sense to disable it.
I'm following up internally on the 5hours long running dataflow. It seems to be a known issue and we'll work on a fix.
Thanks for continuing to report feedback!
- Element115
EDIT::one needs to set a destination and then only disable staging, in order for the the Publish button to not disappear. There is no way to publish if staging is disabled and no destination is set.
Interesting... the Publish button does not disappear if I set the destination before disabling Enable staging. So I tried again but this time the destination is a lakehouse I created just for this. And it does not work. No data, no table created by the DF in the lakehouse. This is with one DF, not the workaround with 2 DFs. Whereas with Enable staging, at least you get the data but not the source column names, which get replaced by generic names such as Column1, Column2, etc....
My network admin did open port 1433 and set the Firewall rule according to the Microwsoft doc for the on-prem Power BI data gateway, so...
Here is the screenshot:
- frlunett
From our telemetry it does seem like a gateway connectivity issue. Can you check the error message that's shown when you click on the Sector_TS_* activity from the refresh history dialog?
We do have a blog post that describes limitations and considerations when using Data Gateway with output destination in the context of Dataflow Gen2: On-premises data gateway considerations for data destinations in Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
Finally, it is by design to disallow publishing a dataflow that only contains queries that are not marked as staged nor have an output destination set.
Thanks,
Frank.
Hello Element115 ,
We haven’t heard from you after the last response by frlunett - and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .
- Element115
By the way, I am trying to repurpose a third dataflow to chain it to a fourth one, so to do the same thing I did with DF1 and DF2 but against my own created lakehouse instead of the staging lakehouse and I come across multiple issues... one of which is the following:
Since DF3 already had a destination set, I delete it, publish, and then re-open DF3 only to see that the destination is set again EVEN THOUGH I deleted it! So to be clear, after choosing Delete in the menu you see in the screenshot, and Publishing the DF, the delete did not have any effect even though the UI showed that the destination got cleared.