Get certified in Microsoft Fabric—for free! For a limited time, the Microsoft Fabric Community team will be offering free DP-600 exam vouchers. Prepare now
In reference to the staging Lakehouse "ghosts" that are created after Dataflow Gen 2 dataflows are used to connect to, transform, and put data into Lakehouse tables.
Apparently these are intended to be hidden but for the time being are visible. See this reddit thread for info on this https://www.reddit.com/r/MicrosoftFabric/comments/13shbex/what_is_dataflowsstaginglakehouse/
When are these created exactly? And also can they be recreated if they are deleted?
While troubleshooting Dataflow Gen 2 dataflow I deleted the Lakehouse table it created and also the "ghost" staging Lakehouse artifacts.
I thought maybe by simply re-running the dataflow it might recreate the staging Lakehouse artificts but nope it didn't do that.
I also tried deleting and recreating the dataflow "Data destination" it might trigger recreating the staging Lakehouse artificts, to no effect either.
It seems I have to delete my dataflows and recreate them from scratch.
Does anyone have any insight into these staging Lakehouse artificts?
Edit to add:
For example, if I delete all of my Dataflow Gen 2 dataflows as part of a revision or troubleshooting, should I also then delete the Lakehouse staging artifacts so they can be recreated "fresh" when I recreate my Dataflow Gen 2 dataflows ? Are the Lakehouse staging artifacts specifically relevant only to the Dataflow Gen 2 dataflows that created them?
From Dataflow Gen 2 Known Issues:
https://learn.microsoft.com/en-us/fabric/data-factory/known-issue-staging-artifact
In a new workspace, when an end-user first launches a Dataflow Gen2, a set of staging artifacts, such as a new Lakehouse and a new Data Warehouse, are automatically created for the whole workspace. These artifacts can be used for a set of functionalities available for Dataflow Gen2.
A user is able to see these staging artifacts in the workspace list and lineage and being able to interact with these artifacts.
If a user deletes or modifies any of these staging artifacts, the workspace can be set in a state that won't be able to use these artifacts because of misconfiguration.
APPLIES TO: ✔️ Dataflow Gen2 in Microsoft Fabric
Status: Open
Problem area: Data Factory
When trying to refresh a Dataflow gen2 and the error refers to:
After deleting the staging artifacts, you're unable to get them back.
Recreate your Dataflow Gen2 solution in a new workspace.
Hello @009co
Thanks for the question and using Fabric community .
I am reaching out to the internal team to get help on the ask .
Thanks
Himanshu
Check out the October 2024 Fabric update to learn about new features.
Learn from experts, get hands-on experience, and win awesome prizes.
User | Count |
---|---|
3 | |
2 | |
1 | |
1 | |
1 |