Forum Discussion
Dataflow Gen2 authentication to Lakehouse without user dependency?
Hi,
We have run into a challenge with Dataflow Gen2 in Microsoft Fabric.
When creating a dataflow and selecting a destination, for example a Lakehouse in the same workspace, the authentication appears to require an Organizational account. In practice, this means that the connection becomes tied to a specific user.
This creates governance and operational challenges in larger organizations. If the user who created or authenticated the dataflow leaves the company, changes role, or loses access, someone else needs to take over the dataflow and re-authenticate the connection. In an organization with thousands of users, this quickly becomes difficult to manage and track.
I understand that in some scenarios it may be possible to use a service principal, but that raises another question: how many service principals should be created and maintained? Should it be one per workspace, per domain, per team, per solution, or something else?
For this specific scenario, Workspace identity feels like a much more natural fit. A workspace-scoped identity would reduce the dependency on individual user accounts and make the solution more robust over time.
Microsoft’s documentation describes Workspace identity as a Fabric-managed service principal associated with a workspace, where Fabric manages the credentials and can use the identity with resources that support Microsoft Entra authentication. The Lakehouse connector documentation also shows that, for Dataflow Gen2, the supported authentication type is currently Organizational account.
My questions are:
- What is the recommended best practice for authenticating Dataflow Gen2 when writing to a Lakehouse in the same workspace, without creating a dependency on a personal user account?
- Is using a service principal the recommended approach today?
- If so, what is the recommended pattern: one service principal per workspace, per team, per solution, or something else?
- Are there any plans to support Workspace identity for this scenario?
It would be very helpful to hear how other organizations are handling this, especially in larger environments where ownership changes and user lifecycle management can otherwise become quite difficult.
Thanks in advance!
Hi Parchitect,
Thank you for reaching out to the Microsoft Fabric Communtiy.
Currently, when Dataflow Gen2 writes to a Lakehouse, only Organizational account authentication is supported, so the connection remains tied to the authenticated user.For this specific scenario, service principal authentication is not available for the Lakehouse destination in Dataflow Gen2. As a workaround, many organizations use a dedicated non-personal service account and manage access through workspace roles to reduce dependency on individual users.
If you require fully non-user-based authentication, consider using Fabric Pipelines or Notebooks, which support service principal authentication for automation scenarios.
Workspace Identity aligns well with this requirement, but it is not currently supported for Lakehouse destination authentication in Dataflow Gen2.
For reference:
- Workspace Identity: https://learn.microsoft.com/fabric/security/workspace-identity
- Dataflow Gen2 destinations: https://learn.microsoft.com/fabric/data-factory/dataflow-gen2-data-destinations
- Service principals in Fabric: https://learn.microsoft.com/fabric/enterprise/service-principals
I hope this helps. Please let us know if you have any further questions.
Hi Parchitect,
Dataflow Gen 2 is inately user bound right now.
THey need to be owned by a user to run, regardless of what connections they're using. As soon as you try to bind the dataflow to a service principal it will no longer be runnable.I hope that this is improved in the future, but it's where we're at right now. I do not recommend using dataflow gen 2 for anything that can't be owned by an individual.
3 Replies
- ParchitectSolution Sage
tayloramy v-anbandari
Thanks for the answers.
Its as i predicted, that only a non-personal service account is the workaround for now. Non-personal service account is not very easy to create and with the restrictions we have in our organisations. - tayloramySuper User
Hi Parchitect,
Dataflow Gen 2 is inately user bound right now.
THey need to be owned by a user to run, regardless of what connections they're using. As soon as you try to bind the dataflow to a service principal it will no longer be runnable.I hope that this is improved in the future, but it's where we're at right now. I do not recommend using dataflow gen 2 for anything that can't be owned by an individual.
- v-anbandariCommunity Support
Hi Parchitect,
Thank you for reaching out to the Microsoft Fabric Communtiy.
Currently, when Dataflow Gen2 writes to a Lakehouse, only Organizational account authentication is supported, so the connection remains tied to the authenticated user.For this specific scenario, service principal authentication is not available for the Lakehouse destination in Dataflow Gen2. As a workaround, many organizations use a dedicated non-personal service account and manage access through workspace roles to reduce dependency on individual users.
If you require fully non-user-based authentication, consider using Fabric Pipelines or Notebooks, which support service principal authentication for automation scenarios.
Workspace Identity aligns well with this requirement, but it is not currently supported for Lakehouse destination authentication in Dataflow Gen2.
For reference:
- Workspace Identity: https://learn.microsoft.com/fabric/security/workspace-identity
- Dataflow Gen2 destinations: https://learn.microsoft.com/fabric/data-factory/dataflow-gen2-data-destinations
- Service principals in Fabric: https://learn.microsoft.com/fabric/enterprise/service-principals
I hope this helps. Please let us know if you have any further questions.