Best practice user-independent connections between Dataflow Gen2 and Lakehouse

[Anonymous]

Hi,
I'm wondering how best to define the connections between Dataflows Gen2 and the corresponding source and sink lakehouses. Currently, the connection is specified via the active developer of the dataflow and is overwritten each time another developer takes over the dataflow and opens it. We would prefer a user-independent solution in which the connection credentials are managed by a service principle or an app registration. My research on the subject didn't turn up anything and the configuration options in Dataflow didn't help either.

In the “Manage Connections” menu for the relevant connection, only an organizational account can be added.

What am I doing wrong or what have I overlooked?

[v-tsaipranay · Accepted solution]

Hi @Anonymous ,

Thank you for posting in the Microsoft Fabric Community.

As mentioned by @R1k91 , the behavior you're encountering is due to a limitation in Dataflows where connection credentials are tied to the Dataflow owner (the active user). Unfortunately, Dataflows do not currently support managing connections through service principals or app registrations. However, to address this, you can consider the following options:

If you are using Azure Data Lake or similar storage systems, you could authenticate using a service principal at the storage level. While Dataflows may not directly support this, you could configure it externally to keep the connection independent of individual users.

Store your connection credentials in Azure Key Vault and reference them within your Dataflow. You can configure a service principal to access the Key Vault, ensuring credentials remain consistent regardless of the developer.

Set up a shared service principal or app registration as the "owner" of the Dataflow, providing consistent access to your sources and sinks without being tied to a specific user.

I hope this will resolve your issue, if you need any further assistance, feel free to reach out.

If this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

Thankyou.

[Anonymous]

Hi,
Thank you for your detailed reply and the suggested solutions. Your steps describe exactly what I am trying to implement, but there still seems to be a problem. Even if the service principle or the app was defined as the owner of the dataflow and the connection is defined by it, a developer who customizes the dataflow would overwrite all credentials again. Either the connection would have to be manually adjusted again after each development step, or all developers would only be allowed to access fabric objects via the stored registration.

Do I see that correctly?

[R1k91]

just to add that AKV support has been announcend at FabConf happening this week.
What's new and planned for Data Factory in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

[R1k91]

as far as i know this is a dataflows limitation.

the connection is owner by the dataflows' owner.