Forum Discussion

lskumardadi's avatar
lskumardadi
New Member
25 days ago
Solved

Choosing connection types for lakehouse

Hello Team,
I am planning to create a connection to lakehouse using connection type as lakehouse but it allows only Oauth in future it will be a problem if the owner is out of organization,If I create the same with type sqlserver then that connection will be generic to specific lakehouse only as it prompts dbname(lakehouse name) while creating connection , in this case the number of connections will increase.So what is the best approach to create connection targetting to lakehouse.

  • ssrithar's avatar
    ssrithar
    25 days ago

    Hi lskumardadi,

    This is a great question, and I think it highlights one of the current design considerations in Fabric.

    Today, Lakehouse connections are scoped to a specific Lakehouse, so if each project owns its own Lakehouse, it's expected that the number of SQL Analytics Endpoint connections will grow. At the moment, there isn't a concept of a single generic SQL connection that can dynamically target multiple Lakehouses.

    From an enterprise perspective, I've generally seen customers follow these practices:

    Use service principals or managed identities (where supported) instead of personal OAuth accounts for production workloads.
    Treat connections as environment infrastructure (DEV/QA/PROD) and provision them through CI/CD or Infrastructure-as-Code, rather than creating and maintaining them manually.
    Maintain one connection per Lakehouse only when the Lakehouse represents a separate security or ownership boundary. Although this increases the connection count, it provides clear governance, auditing, and least-privilege access.

    While having 15, 50, or even 100+ connections may initially seem excessive, the operational overhead becomes much smaller once connection creation and deployment are automated.

     

    If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
    If I misunderstand your needs or you still have problems on it, please feel free to let me know. Thanks a lot!

8 Replies

  • Hi lskumardadi ,

     

    The choice largely depends on your organization's authentication and governance strategy:

     

    Lakehouse connection (OAuth): This is the native Fabric connection and is the recommended option when using user identity. However, as you mentioned, if the connection is tied to an individual user's OAuth credentials, it can become an operational risk if that user leaves the organization or their account is disabled.


    SQL Server connection: This provides more flexibility and can use service principals or SQL authentication (where supported), but it requires specifying the Lakehouse (database) name. If you have many Lakehouses, you'll end up managing many separate connections, which increases administrative overhead.

     

    For enterprise environments, a common approach is:

     

    Use service principals or managed identities wherever Fabric supports them, instead of individual user accounts.
    Store credentials centrally (for example, in Azure Key Vault where applicable) and automate connection deployment through CI/CD.


    Accept multiple SQL connections if they are required, but manage them as infrastructure rather than manually. The additional connections are usually easier to maintain than connections that depend on a specific user's identity.

     

    If you're using Data Factory or Fabric Pipelines, another consideration is whether your workloads can reference workspace items directly rather than relying on individual connections, as this reduces connection management.

     

    If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
    If I misunderstand your needs or you still have problems on it, please feel free to let me know. Thanks a lot!

    • lskumardadi's avatar
      lskumardadi
      New Member

      Hello,

      Thank you for your prompt response.

      I understand the recommendation to use Service Principal authentication through a SQL Server/SQL Analytics Endpoint connection for production workloads. However, in our scenario we have multiple data projects, and each project has its own dedicated Lakehouse.

      If we use SQL Server/SQL Analytics Endpoint connections, the connection is tied to a specific Lakehouse (database). As a result, the number of connections grows significantly across environments.

      For example:

      • Dev: 5 Lakehouses = 5 connections
      • QA: 5 Lakehouses = 5 connections
      • Prod: 5 Lakehouses = 5 connections

      This results in a total of 15 connections to manage, and the number will continue to increase as more projects and Lakehouses are added.

      Given this scenario, what would be the Microsoft-recommended best practice?

      • Should we continue with SQL Analytics Endpoint connections and manage the larger number of connections through automation/CI-CD?
      • Or is there a recommended way to create Lakehouse connections that are not dependent on an individual user's OAuth identity (for example, using a service account, managed identity, or service principal)?
      • How do customers with a large number of Lakehouses typically handle connection management in Fabric?

      We are trying to balance governance and service account independence against the operational overhead of managing a growing number of Lakehouse-specific connections.

      Thanks in advance for your guidance.

      • ssrithar's avatar
        ssrithar
        Super User

        Hi lskumardadi,

        This is a great question, and I think it highlights one of the current design considerations in Fabric.

        Today, Lakehouse connections are scoped to a specific Lakehouse, so if each project owns its own Lakehouse, it's expected that the number of SQL Analytics Endpoint connections will grow. At the moment, there isn't a concept of a single generic SQL connection that can dynamically target multiple Lakehouses.

        From an enterprise perspective, I've generally seen customers follow these practices:

        Use service principals or managed identities (where supported) instead of personal OAuth accounts for production workloads.
        Treat connections as environment infrastructure (DEV/QA/PROD) and provision them through CI/CD or Infrastructure-as-Code, rather than creating and maintaining them manually.
        Maintain one connection per Lakehouse only when the Lakehouse represents a separate security or ownership boundary. Although this increases the connection count, it provides clear governance, auditing, and least-privilege access.

        While having 15, 50, or even 100+ connections may initially seem excessive, the operational overhead becomes much smaller once connection creation and deployment are automated.

         

        If this post helps, then please appreciate giving a Kudos or accepting as a Solution to help the other members find it more quickly.
        If I misunderstand your needs or you still have problems on it, please feel free to let me know. Thanks a lot!

  • v-csrikanth's avatar
    v-csrikanth
    Community Support

    Hi lskumardadi 
    We would like to inquire whether have you got the chance to check the solutions provided by samchandel ssrithar ShivekMaharaj in community to resolve the issue. We hope the information provided helps to clear the query. Should you have any further queries, kindly feel free to contact the Microsoft Fabric community.

  • v-csrikanth's avatar
    v-csrikanth
    Community Support

    Hi lskumardadi 

    We wanted to follow up and check whether you had a chance to review our previous response and if the issue has been resolved.

    If you are still experiencing the issue or have any additional questions, please feel free to share an update. We’ll be happy to assist further.

    Thank you,
    C Srikanth
    Community Support Team

  • Use a dedicated service account/service principal for the Lakehouse OAuth connection instead of an individual user. This avoids ownership issues when someone leaves and also prevents creating separate SQL connections for every Lakehouse.

  • ShivekMaharaj's avatar
    ShivekMaharaj
    Impactful Individual

    Hi lskumardadi,

     

    I think the key point here is that the connection should normally be designed around the shared security and access boundary, not automatically around each individual project.

     

    If all five projects need the same type of access to the same Lakehouse, I would avoid creating five separate connections unless there is a genuine need for separate credentials, ownership or isolation.

     

    A practical approach would be:

    • Use a shared organisational or service identity rather than individual user OAuth for production workloads.
    • Reuse the same connection where the projects have the same permissions and lifecycle.
    • Create separate connections only where projects require different identities, different access levels, separate ownership or stronger isolation.
    • Apply a consistent naming convention so connections remain easy to identify as the environment grows.

    For example, instead of creating one connection per project, you could define connections by environment and access pattern, such as:

    • DEV - Lakehouse - ReadWrite
    • TEST - Lakehouse - ReadWrite
    • PROD - Lakehouse - ReadOnly

    If a particular project requires elevated access or a different identity, then that project can have its own dedicated connection.

     

    I would also avoid relying on a personal OAuth connection for long-running production workloads, because it creates a dependency on an individual account and can become harder to manage when ownership changes.

     

    So in your case, I would not assume that five Lakehouses must always mean five completely separate connection strategies. I would first group them by:

    • Environment
    • Required permissions
    • Authentication identity
    • Ownership and support responsibility

    Then create only the minimum number of connections needed to represent those boundaries.

     

    The main question is therefore not really “how many projects do I have?”, but “how many distinct security and operational boundaries do I need to manage?”

  • samchandel's avatar
    samchandel
    Regular Visitor

    Hi  ,

    You've basically laid out the two real trade-offs correctly, so let me try to add some clarity on picking between them.

    The Lakehouse connector currently only supports OAuth 2.0 (organizational account), service principal isn't available on that connector surface yet. So your concern about the owner leaving the org is valid, that connection is tied to a real user identity and will break if that account gets disabled or loses access.

    The SQL analytics endpoint connection (using SQL Server as the connection type) does support Service Principal, which solves the "owner leaves the org" problem since it's not tied to any individual. Your point about it being specific to one Lakehouse (needing the dbname prompted at creation) is accurate too, there isn't a generic "connect to any Lakehouse" option at the SQL endpoint level, each connection is scoped to a specific database.

    Given those constraints, the common pattern is:

    If you need broad reuse across many Lakehouses and can tolerate OAuth tied to an owner, the Lakehouse connector is simpler to set up, but you'd want that owner to be a dedicated service account (not a real employee's account) so it's not tied to any one person leaving.

    If long-term stability and not depending on any user identity matters more, go with the SQL endpoint + Service Principal route, even though it means one connection per Lakehouse. The "more connections" concern is real but it's a manageable, static list versus a connection that can silently break when someone leaves.

    A hybrid a lot of teams land on is creating a dedicated non-human service account, using that as the "owner" for Lakehouse connector connections where SPN isn't available, and reserving actual Service Principal auth for the workspaces/endpoints where it's supported (SQL endpoint, Warehouse, Pipelines). That way you're not tied to any one person either way, you're just choosing between OAuth-as-a-service-account and native SPN depending on which surface you're connecting through.

    If minimizing the number of connections matters more to you than authentication type, the Lakehouse connector with a service account as owner is probably your best bet. If avoiding any user-identity dependency entirely is the priority, the SQL endpoint with Service Principal is worth the extra per-Lakehouse setup.