Forum Discussion

shivangjha's avatar
shivangjha
Regular Visitor
2 years ago
Solved

Copy Activity: Fabric + Azure Key Vault + external database connection

<UPDATE> - This is in the context of Fabric Data Factory Copy 
I have a use case for the data pipeline with Copy Activity.

Current State

The Copy activity utilises mongoDB Atlas for pipelines connector plugin (!?) to copy data from mongodb cluster to Fabric Lakehouse.

For test purposes, i had created this connection by providing servers and credentials explicitely:

and that all works fine until I want to deploy this to another environement, say Prod, identified by a different workspace.

 

Expected target state

I want the credentials to be fetched from Azure Key Vault and those to be utilised in the data pipeline copy activity without manual intervention.
Is it feasible?

I understand that as a solution/work around, I can configure a web activity in the data pipeline to get secrets/credentials from Azure Key Vault - but the real issue lies in how to use that in the copy activity/ creating a connection to the mongo source (which would be a PROD mongo db in this case).

Any help is deeply appreciated.
Would be happy to provide more details, if needed.

Need Help
Pipelines
Show and Tell

6 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi shivangjha

     

    It is feasible to use Azure Key Vault to extract credentials and use them in replication activities for data pipelines.

     

    You can try the following steps:

     

    Set the managed identity for the data factory. Ensure that the data factory has a managed identity associated with it.

     

    Configure Azure Key Vault. Open Key Vault's access policy and add a hosting identity with permissions and secrets.

     

    Store secrets in Azure Key Vault. Navigate to Key Vault and store MongoDB credentials as confidential.

     

    Add a Web activity to get secrets. In the data factory pipeline, add a new Web activity to extract secrets from the Azure Key Vault.

     

    Use of secrets in replication activities. In the Copy activity, dynamic content is used to reference the output of the Web activity.

     

    You can view the link below for more details:

     

    Use Azure Key Vault secrets in pipeline activities - Azure Data Factory | Microsoft Learn

     

    Store credentials in Azure Key Vault - Azure Data Factory | Microsoft Learn

     

    Regards, 

    Nono Chen 

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly. 

  • shivangjha's avatar
    shivangjha
    Regular Visitor
    2 years ago

    Hi Anonymous ,

    Thanks for your reply.
    It seems that this solution is relevant to Azure data factory pipelines.

    I am looking to achieve this is Fabric Data Factory pipelines.. 
    Apologies, if it was not that clear in my original post. Have updated my original post too.

    Thanks

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi shivangjha 

     

    Can you tell me if your problem is solved? If yes, please accept it as solution.

     

    Regards,

    Nono Chen

    • shivangjha's avatar
      shivangjha
      Regular Visitor
      2 years ago

      Hi Anonymous ,
      Thanks for your update on the procedure  - how to get azure key vault secrets.
      It does not solve the actual problem of using those secrets in the copy activity in fabric. 
      As per my original post, i had specifically requested some info on how to use azure key vault secrets in the Copy data activity.

      excerpt from the original post
      "
      I understand that as a solution/work around, I can configure a web activity in the data pipeline to get secrets/credentials from Azure Key Vault - but the real issue lies in how to use that in the copy activity/ creating a connection to the mongo source (which would be a PROD mongo db in this case)."

      If you can help on those lines, it would be super cool.

      Appreciate your efforts though 🙂

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hi shivangjha 

         

        In response to this sentence: “but the real issue lies in how to use that in the copy activity/ creating a connection to the mongo source (which would be a PROD mongo db in this case).”

         

        First, you store MongoDB's connection string, or credentials, in Azure Key Vault. Make sure you have access to those secrets.

         

        In the data factory pipeline, add a new Web activity. Configure this activity to get secrets from Azure Key Vault. You will need to provide the Key Vault URL and secret name.

         

        Make sure your data factory has access to Azure Key Vault.

         

        Solved: Accessing Azure KeyVault secret for Fabric CopyDat... - Microsoft Fabric Community

         

        After obtaining secrets in a Web activity, you can use them in a replication activity. You can set MongoDB's connection string by referencing the output of your Web activity. For example:

         

        @activity('WebActivityName').output.value

         

        Regards,

        Nono Chen