topic Re: Secure Key Vault access without Workspace Identity in Data Engineering

Secure Key Vault access without Workspace Identity

Eddykleinjan — Sat, 07 Sep 2024 13:08:12 GMT

Hi,

I'm ready to dive into the first real Fabric project getting data from a specific app's API. This requires using secrets to access the soure data app's API. I don't want to store these secrets in source code (insecure coding) and want to use Azure Key Vault instead. To access the Azure Key vault, the notebook process needs to have access to Azure Key Vault.

Now Fabric has an option to assign a identity to a workspace (workspace identity) and authorize that workspace identity to access the key vault. This souds like a solution, but the catch here is that the workspace identity is only available starting with the Fabric F64 capacity. Since pricing for F64 starts at USD 10k+ per month, that is not an option for us or our customers.

Also note that I would like to run this code (notebook) unattended, so an interactive authentication of the user running the notebook is not an option.

Anyone ran into this? Would love to hear what approach you took.

Re: Secure Key Vault access without Workspace Identity

codenamesql — Sun, 08 Sep 2024 21:19:48 GMT

Hey @Eddykleinjan, The F64 + requirement is no longer valid for Workspace Identity. You can now create a workspace identity with any F sku. This should allow you to accomplish what you are looking for.

Re: Secure Key Vault access without Workspace Identity

Eddykleinjan — Mon, 09 Sep 2024 07:23:24 GMT

That's good news! Should then work ask hoped for/expected. I'll give it a try.

Re: Secure Key Vault access without Workspace Identity

Anonymous — Mon, 09 Sep 2024 08:08:17 GMT

Hi @Eddykleinjan

I haven't tested whether the workspace identity can access the Azure Key Vault. Could you please share your results with us later? Thank you in advance.

Best Regards,
Jing

Re: Secure Key Vault access without Workspace Identity

Eddykleinjan — Mon, 09 Sep 2024 08:45:21 GMT

Hi Jing,
Will do! For now creating a workspace identity gives an error (status: Failed). Will start a support incident on this error occurs both under a Fabric Trial capacity and a real Fabric F2 capacity.
Best regard, Eddy

Re: Secure Key Vault access without Workspace Identity

Jaimini — Mon, 09 Sep 2024 15:09:32 GMT

Hi @Eddykleinjan

The F64+ requirement is no longer valid. You can now create up to 1,000 workspace identities.
Important: Currently, authentication with workspace identities, specifically for Key Vault, is not available.
Interestingly, if you have access to Key Vault with your domain account and run the notebook using the same account, you will be able to access the Key Vault secrets. You can use mssparkutils for this.
Additionally, you can create a Spark job definition using the same code (as mentioned in point 3). If you schedule and run it, it will work, using your account for authentication.

For now, managed/workspace identity authentication for Key Vault is not functioning.

@Anonymous

Re: Secure Key Vault access without Workspace Identity

frithjof_v — Mon, 09 Sep 2024 17:24:45 GMT

Perhaps you can use this:

https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities#credentials-utilities

Re: Secure Key Vault access without Workspace Identity

Eddykleinjan — Tue, 10 Sep 2024 15:09:24 GMT

Thanks for the pointer, that works! It reads the value from the Key Vault both when run interactively and when running the notebook scheduled.

When running the notebook scheduled, it seems to run under the account that has created the schedule for the notebook. That way the scheduled notebook run could read the secret from the Azure Key Vault.

test_secret = notebookutils.credentials.getSecret('https://<url_to_key_vault>/', '<secret_name>') # Reverse the value in order to show it. Otherwise it will be shown as '[REDACTED]' reversed_string = test_secret[::-1] print (f"Secret value reversed: {reversed_string}")

Re: Secure Key Vault access without Workspace Identity

Eddykleinjan — Tue, 10 Sep 2024 15:14:44 GMT

Hi Jing,

Creating the workspace identity didn't work because the workspace name had special characters in the name, a space in my case. Microsoft had identified this as a problem and will fix this.

@Jaimini reported that accessing the Key Vault is not possible using the workspace identity, I chose to use the notebookutils way that @frithjof_v adviced. That worked; see my reply to his message.

Re: Secure Key Vault access without Workspace Identity

Anonymous — Wed, 11 Sep 2024 08:15:11 GMT

Thank you very much! I really appreciate the results you share!