Lakehouse staging "ghosts" - intended to be hidden but show when dataflow gen 2 dataflows created

[009co]

In reference to the staging Lakehouse "ghosts" that are created after Dataflow Gen 2 dataflows are used to connect to, transform, and put data into Lakehouse tables. 

Apparently these are  intended to be hidden but for the time being are visible. See this reddit thread for info on this https://www.reddit.com/r/MicrosoftFabric/comments/13shbex/what_is_dataflowsstaginglakehouse/

When are these created exactly? And also can they be recreated if they are deleted?

While troubleshooting Dataflow Gen 2 dataflow I deleted the Lakehouse table it created and also the "ghost" staging Lakehouse artifacts. 

I thought maybe by simply re-running the dataflow it might recreate the staging Lakehouse artificts but nope it didn't do that.

I also tried deleting and recreating the dataflow "Data destination" it might trigger recreating the  staging Lakehouse artificts, to no effect either.

It seems I have to delete my dataflows and recreate them from scratch.

Does anyone have any insight into these  staging Lakehouse artificts? 

Edit to add:

For example, if I delete all of my Dataflow Gen 2 dataflows as part of a revision or troubleshooting, should I also then delete the Lakehouse staging artifacts so they can be recreated "fresh" when I recreate my  Dataflow Gen 2 dataflows ? Are the Lakehouse staging artifacts specifically relevant only to the Dataflow Gen 2 dataflows that created them?

[009co]

From Dataflow Gen 2 Known Issues:

https://learn.microsoft.com/en-us/fabric/data-factory/known-issue-staging-artifact

Known issue - Staging artifacts aren't hidden and can be modified by the end-user

• Article
• 06/06/2023
• 2 contributors

Feedback

In this article

1. Symptoms
2. Solutions and workarounds

In a new workspace, when an end-user first launches a Dataflow Gen2, a set of staging artifacts, such as a new Lakehouse and a new Data Warehouse, are automatically created for the whole workspace. These artifacts can be used for a set of functionalities available for Dataflow Gen2.

A user is able to see these staging artifacts in the workspace list and lineage and being able to interact with these artifacts.

If a user deletes or modifies any of these staging artifacts, the workspace can be set in a state that won't be able to use these artifacts because of misconfiguration.

APPLIES TO: ✔️ Dataflow Gen2 in Microsoft Fabric

Status: Open

Problem area: Data Factory

Symptoms

When trying to refresh a Dataflow gen2 and the error refers to:

• null Error: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: PowerBIEntityNotFound: PowerBIEntityNotFound

After deleting the staging artifacts, you're unable to get them back.

Solutions and workarounds

Recreate your Dataflow Gen2 solution in a new workspace.

[HimanshuS-msft]

Hello @009co 

Thanks for the question and using Fabric community . 
I am reaching out to the internal team to get help on the ask .

Thanks 
Himanshu