topic Lakehouse staging "ghosts" - intended to be hidden but show when dataflow gen 2 dataflows created in Fabric platform

Lakehouse staging "ghosts" - intended to be hidden but show when dataflow gen 2 dataflows created

009co — Thu, 01 Jun 2023 08:48:13 GMT

In reference to the staging Lakehouse "ghosts" that are created after Dataflow Gen 2 dataflows are used to connect to, transform, and put data into Lakehouse tables.

Apparently these are intended to be hidden but for the time being are visible. See this reddit thread for info on this https://www.reddit.com/r/MicrosoftFabric/comments/13shbex/what_is_dataflowsstaginglakehouse/

When are these created exactly? And also can they be recreated if they are deleted?

While troubleshooting Dataflow Gen 2 dataflow I deleted the Lakehouse table it created and also the "ghost" staging Lakehouse artifacts.

I thought maybe by simply re-running the dataflow it might recreate the staging Lakehouse artificts but nope it didn't do that.

I also tried deleting and recreating the dataflow "Data destination" it might trigger recreating the staging Lakehouse artificts, to no effect either.

It seems I have to delete my dataflows and recreate them from scratch.

Does anyone have any insight into these staging Lakehouse artificts?

Edit to add:

For example, if I delete all of my Dataflow Gen 2 dataflows as part of a revision or troubleshooting, should I also then delete the Lakehouse staging artifacts so they can be recreated "fresh" when I recreate my Dataflow Gen 2 dataflows ? Are the Lakehouse staging artifacts specifically relevant only to the Dataflow Gen 2 dataflows that created them?

Re: Lakehouse staging "ghosts" - intended to be hidden but show when dataflow gen 2 dataflows created

HimanshuS-msft — Thu, 08 Jun 2023 05:17:12 GMT

Hello 009co

Thanks for the question and using Fabric community .
I am reaching out to the internal team to get help on the ask .

Thanks
Himanshu

Re: Lakehouse staging "ghosts" - intended to be hidden but show when dataflow gen 2 dataflows created

009co — Sat, 10 Jun 2023 15:40:22 GMT

From Dataflow Gen 2 Known Issues:

https://learn.microsoft.com/en-us/fabric/data-factory/known-issue-staging-artifact

Known issue - Staging artifacts aren't hidden and can be modified by the end-user

Article
06/06/2023
2 contributors

Feedback

Symptoms

When trying to refresh a Dataflow gen2 and the error refers to:

null Error: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: PowerBIEntityNotFound: PowerBIEntityNotFound

After deleting the staging artifacts, you're unable to get them back.

Solutions and workarounds

Recreate your Dataflow Gen2 solution in a new workspace.

topic Lakehouse staging "ghosts" - intended to be hidden but show when dataflow gen 2 dataflows created in Fabric platform

Lakehouse staging "ghosts" - intended to be hidden but show when dataflow gen 2 dataflows created

Re: Lakehouse staging "ghosts" - intended to be hidden but show when dataflow gen 2 dataflows created

Re: Lakehouse staging "ghosts" - intended to be hidden but show when dataflow gen 2 dataflows created

Known issue - Staging artifacts aren't hidden and can be modified by the end-user

In this article

Symptoms

Solutions and workarounds