topic Re: Copy Activity: Fabric + Azure Key Vault + external database connection in Pipelines

Copy Activity: Fabric + Azure Key Vault + external database connection

shivangjha — Wed, 03 Jul 2024 07:26:37 GMT

<UPDATE> - This is in the context of Fabric Data Factory Copy
I have a use case for the data pipeline with Copy Activity.

Current State

The Copy activity utilises mongoDB Atlas for pipelines connector plugin (!?) to copy data from mongodb cluster to Fabric Lakehouse.

For test purposes, i had created this connection by providing servers and credentials explicitely:

and that all works fine until I want to deploy this to another environement, say Prod, identified by a different workspace.

Expected target state

I want the credentials to be fetched from Azure Key Vault and those to be utilised in the data pipeline copy activity without manual intervention.
Is it feasible?

I understand that as a solution/work around, I can configure a web activity in the data pipeline to get secrets/credentials from Azure Key Vault - but the real issue lies in how to use that in the copy activity/ creating a connection to the mongo source (which would be a PROD mongo db in this case).

Any help is deeply appreciated.
Would be happy to provide more details, if needed.

Re: Copy Activity: Fabric + Azure Key Vault + external database connection

Anonymous — Wed, 03 Jul 2024 07:13:46 GMT

Hi shivangjha

It is feasible to use Azure Key Vault to extract credentials and use them in replication activities for data pipelines.

You can try the following steps:

Set the managed identity for the data factory. Ensure that the data factory has a managed identity associated with it.

Configure Azure Key Vault. Open Key Vault's access policy and add a hosting identity with permissions and secrets.

Store secrets in Azure Key Vault. Navigate to Key Vault and store MongoDB credentials as confidential.

Add a Web activity to get secrets. In the data factory pipeline, add a new Web activity to extract secrets from the Azure Key Vault.

Use of secrets in replication activities. In the Copy activity, dynamic content is used to reference the output of the Web activity.

You can view the link below for more details:

Use Azure Key Vault secrets in pipeline activities - Azure Data Factory | Microsoft Learn

Store credentials in Azure Key Vault - Azure Data Factory | Microsoft Learn

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Copy Activity: Fabric + Azure Key Vault + external database connection

shivangjha — Wed, 03 Jul 2024 07:27:20 GMT

Hi Anonymous ,

Thanks for your reply.
It seems that this solution is relevant to Azure data factory pipelines.

I am looking to achieve this is Fabric Data Factory pipelines..
Apologies, if it was not that clear in my original post. Have updated my original post too.

Thanks

Re: Copy Activity: Fabric + Azure Key Vault + external database connection

Anonymous — Wed, 03 Jul 2024 07:53:19 GMT

Hi shivangjha

I am aware of the problem. And I updated the following link:

Get Azure Key Vault secrets in Fabric pipeline – Simplyfies

Regards,

Nono Chen

Re: Copy Activity: Fabric + Azure Key Vault + external database connection

Anonymous — Tue, 16 Jul 2024 08:05:32 GMT

Hi shivangjha

Can you tell me if your problem is solved? If yes, please accept it as solution.

Regards,

Nono Chen

Re: Copy Activity: Fabric + Azure Key Vault + external database connection

shivangjha — Fri, 19 Jul 2024 09:47:22 GMT

Hi Anonymous ,
Thanks for your update on the procedure - how to get azure key vault secrets.
It does not solve the actual problem of using those secrets in the copy activity in fabric.
As per my original post, i had specifically requested some info on how to use azure key vault secrets in the Copy data activity.

excerpt from the original post
"
I understand that as a solution/work around, I can configure a web activity in the data pipeline to get secrets/credentials from Azure Key Vault - but the real issue lies in how to use that in the copy activity/ creating a connection to the mongo source (which would be a PROD mongo db in this case)."

If you can help on those lines, it would be super cool.

Appreciate your efforts though 🙂

Re: Copy Activity: Fabric + Azure Key Vault + external database connection

Anonymous — Mon, 22 Jul 2024 02:56:26 GMT

Hi shivangjha

In response to this sentence: “but the real issue lies in how to use that in the copy activity/ creating a connection to the mongo source (which would be a PROD mongo db in this case).”

First, you store MongoDB's connection string, or credentials, in Azure Key Vault. Make sure you have access to those secrets.

In the data factory pipeline, add a new Web activity. Configure this activity to get secrets from Azure Key Vault. You will need to provide the Key Vault URL and secret name.

Make sure your data factory has access to Azure Key Vault.

Solved: Accessing Azure KeyVault secret for Fabric CopyDat... - Microsoft Fabric Community

After obtaining secrets in a Web activity, you can use them in a replication activity. You can set MongoDB's connection string by referencing the output of your Web activity. For example:

@activity('WebActivityName').output.value

Regards,

Nono Chen