Forum Discussion
Dataflow Gen2 staging vs destination: why is data only accessible by creator or Lakehouse?
I’m trying to understand the access and storage behaviour of Dataflow Gen2 in Microsoft Fabric.
When I create a Gen2 dataflow with staging enabled and no destination, I can see and work with the dataflow from Power BI Desktop, but workspace members can only see its name; they cannot access or preview any data (dataflow connector).
As I understood, because data is stored in internal OneLake staging, other workspace users don't have rights to see the data inside?
When I configure a Lakehouse as a destination, then:
- Data is written to Lakehouse tables
- Access is only possible via the Lakehouse
- The dataflow itself no longer seems to have data - now not even I can see it from Power BI Desktop with the dataflow connector.
My questions:
- Why is staging-only data not accessible to workspace members even if they can see the dataflow and even if I as a creator can see it via the dataflow connector in Power BI Desktop?
- Why does adding a Lakehouse destination move access entirely away from the dataflow?
Is there Microsoft documentation that explains this bit? Because I can't find these details specifically, any answers would be appreciated.
Here is more information on when moving from Gen1 to Gen2 https://learn.microsoft.com/en-us/fabric/data-factory/move-dataflow-gen1-to-dataflow-gen2 and here are more details: https://community.fabric.microsoft.com/t5/Service/Migration-Dataflows-Gen1-to-Gen2-with-connected-Semantic-models/m-p/4848892
Hi Mesalome
Thank you for reaching out to the Microsoft Fabric Forum Community.
You’re right in your observation and this is not clearly explained end-to-end in a single place in the current documentation.
However, there is partial official documentation. Please take a look; it might be helpful to you. Data in Dataflow Gen2 staging - Microsoft Fabric | Microsoft Learn
If there are any deviations from your expectation please let us know we are happy to address.
Thanks.
8 Replies
- GilbertQ
Super User
Hi Mesalome
When you enable the staging it is a working area for the Dataflow Gen2. The working area is used for the steps in your dataflow. Once the work is complete the staging is no longer needed (typically because the data is written to a data destination)
A Dataflow Gen2 has to have a data destination for other users to view the output. That is why they only see it when it is in a Lakehouse table.
- tayloramy
Hi Mesalome,
Unlike dataflow gen 1, dataflow gen 2 is not a data storage solution, so data is not stored in the dataflow itself.
You need to set a target for the data to be stored.
Don't rely on the internal staging lakehouse/warehouse as those can change and in the past have had changes which break anything external you do with them.
I recommend you send the data to a lakehouse and then pull it into Power BI from there.
v-priyankataCommunity Support
Thank you for the response and for the support from the community.
I would really appreciate it if someone could provide links to the official documentation, as I need a reliable source to reference and validate the suggested characteristics of Gen2.
At the moment, I am struggling to find clear documentation that outlines these details. It feels like this information should be formally documented somewhere, so having direct references would be extremely helpful.
Thank you in advance for any guidance.
- GilbertQ
Here is more information on when moving from Gen1 to Gen2 https://learn.microsoft.com/en-us/fabric/data-factory/move-dataflow-gen1-to-dataflow-gen2 and here are more details: https://community.fabric.microsoft.com/t5/Service/Migration-Dataflows-Gen1-to-Gen2-with-connected-Semantic-models/m-p/4848892
- v-priyankata
Hi Mesalome
Thank you for reaching out to the Microsoft Fabric Forum Community.
You’re right in your observation and this is not clearly explained end-to-end in a single place in the current documentation.
However, there is partial official documentation. Please take a look; it might be helpful to you. Data in Dataflow Gen2 staging - Microsoft Fabric | Microsoft Learn
If there are any deviations from your expectation please let us know we are happy to address.
Thanks.
- v-priyankata
Hi Mesalome
Hope everything’s going smoothly on your end. I wanted to check if the issue got sorted. if you have any other issues please reach community.
