Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn a 50% discount on the DP-600 certification exam by completing the Fabric 30 Days to Learn It challenge.

Reply
009co
Helper IV
Helper IV

Lakehouse staging "ghosts" - intended to be hidden but show when dataflow gen 2 dataflows created

In reference to the staging Lakehouse "ghosts" that are created after Dataflow Gen 2 dataflows are used to connect to, transform, and put data into Lakehouse tables. 

 

Apparently these are  intended to be hidden but for the time being are visible. See this reddit thread for info on this https://www.reddit.com/r/MicrosoftFabric/comments/13shbex/what_is_dataflowsstaginglakehouse/

 

When are these created exactly? And also can they be recreated if they are deleted?

 

While troubleshooting Dataflow Gen 2 dataflow I deleted the Lakehouse table it created and also the "ghost" staging Lakehouse artifacts. 

 

I thought maybe by simply re-running the dataflow it might recreate the staging Lakehouse artificts but nope it didn't do that.

 

I also tried deleting and recreating the dataflow "Data destination" it might trigger recreating the  staging Lakehouse artificts, to no effect either.

 

It seems I have to delete my dataflows and recreate them from scratch.

 

Does anyone have any insight into these  staging Lakehouse artificts? 

 

Edit to add:

 

For example, if I delete all of my Dataflow Gen 2 dataflows as part of a revision or troubleshooting, should I also then delete the Lakehouse staging artifacts so they can be recreated "fresh" when I recreate my  Dataflow Gen 2 dataflows ? Are the Lakehouse staging artifacts specifically relevant only to the Dataflow Gen 2 dataflows that created them?

2 REPLIES 2
009co
Helper IV
Helper IV

From Dataflow Gen 2 Known Issues:

 

 

 

https://learn.microsoft.com/en-us/fabric/data-factory/known-issue-staging-artifact

 

Known issue - Staging artifacts aren't hidden and can be modified by the end-user

  • Article
  • 06/06/2023
  • 2 contributors
Feedback
 

In this article

  1. Symptoms
  2. Solutions and workarounds

In a new workspace, when an end-user first launches a Dataflow Gen2, a set of staging artifacts, such as a new Lakehouse and a new Data Warehouse, are automatically created for the whole workspace. These artifacts can be used for a set of functionalities available for Dataflow Gen2.

A user is able to see these staging artifacts in the workspace list and lineage and being able to interact with these artifacts.

If a user deletes or modifies any of these staging artifacts, the workspace can be set in a state that won't be able to use these artifacts because of misconfiguration.

APPLIES TO: ✔️ Dataflow Gen2 in Microsoft Fabric

Status: Open

Problem area: Data Factory

Symptoms

When trying to refresh a Dataflow gen2 and the error refers to:

  • null Error: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: PowerBIEntityNotFound: PowerBIEntityNotFound

After deleting the staging artifacts, you're unable to get them back.

Solutions and workarounds

Recreate your Dataflow Gen2 solution in a new workspace.

HimanshuS-msft
Community Support
Community Support

Hello @009co 

Thanks for the question and using Fabric community . 
I am reaching out to the internal team to get help on the ask .

Thanks 
Himanshu 

 

 

Helpful resources

Announcements
RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

Expanding the Synapse Forums

New forum boards available in Synapse

Ask questions in Data Engineering, Data Science, Data Warehouse and General Discussion.

MayFabricCarousel

Fabric Monthly Update - May 2024

Check out the May 2024 Fabric update to learn about new features.

LearnSurvey

Fabric certifications survey

Certification feedback opportunity for the community.

Top Solution Authors