Enabling CMK on Fabric Workspaces

[mortoman]

Hi all,

I'm trying to enable CMK on my Fabric tenant by following the Microsoft guide Customer-managed keys for Fabric workspaces - Microsoft Fabric | Microsoft Learn. I have done the following:

Fabric Tenant enabled for CMK, 

Tenant is licensed not trial,

New Key Vault with soft delete and purge protection enabled,

Tenant and key vault in same region,

Created and granted the Fabric Platform CMK app service principal the RBAC, Key Vault Crypto Service Encryption User on the new vault,

Created a new key, copied the key id,

On Fabric created a new empty workspace (no unsupported CMK items in the workspace)

Selected encryption, entered the location of the key ID and tried to enable on the new workspace,

I get the following message, that I can't find any more details than just this:

Anyone else ever get this message and find the issue please ?

thanks mortoman.

[igm87]

We have been provided with the following RCA from Microsoft this morning:

The issue is occurring at AKV's KeyClient.GetKeyAsync API to get Key metadata. Both platform and AKV Product team are jointly working on the fix. Hard validation occurring at KeyProperties.RecoveryLevel property is under the fix. Considering this under preview tag, unfortunately we could not provide any ETA on the fix now.

So there is (as suspected) a fundamental issue with CMK on Fabric at the moment. As this is a Preview feature I guess we just have to wait until it is addressed.

[v-prasare]

@mortoman, thanks for the update here. Can I close this thread for now or do you want to continue posting updates here? Can you confirm on this.

Thanks,

Prashanth

[igm87]

You can close the thread. I'll re-open as and when MS get back to me and let ne know the bug has been fixed.

[v-prasare]

Hi @mortoman2,

Is there any update on your issue from support team. Please do post your progress here. this will help other community members with similar issues

Thanks,

prashanth

[igm87]

No updates as yet, Microsoft support watched me demonstrate the issue. We checked everything, enabled log analytics and as stated previously, we got nothing in the logs!! We held another meeting with the Key Vault SMEs invited, this time we used the exact same key we setup to test CMK via TDE on a SQL managed instance. The SQL managed instance key wrap and key unwrap events were logged but nothing from Fabric. So the issue occurs immediately; even the previously working Fabric capacity WS has now failed with a warning that the key provided is invalid (the same key that was working in the screen shot below!!!). There appears to be something fundamentally wrong with CMK on Fabric at present, perhaps this is what you get for using a preview feature and should be expected I guess.

Will update you as soon as Microsoft support get back to me with an update / plan.   

[v-prasare]

We would like to confirm if our community members answer resolves your query or if you need further help. If you still have any questions or need more support, please feel free to let us know. We are happy to help you.

If we don’t hear back, we’ll go ahead and close this thread. For any further discussions or questions, please start a new thread in the Microsoft Fabric Community Forum we’ll be happy to assist.
Thank you for being part of the Microsoft Fabric Community.

[mortoman2]

Hi, apologies I've had problems getting into my original mortoman account, hence the second account.

Unfortunately the issue isn't resolved yet. I did try (as per below thread) the suggestion from JMCK, but as stated didn't fix the issue. I have a support call with MS today and if we get to the bottom of the issue I'll post an update (using this account now) ;-\ 

[v-prasare]

Thanks for the update @mortoman2. please do post updates here

[v-prasare]

Hi @mortoman,

We would like to follow up to see if your query got resolved? Please let us know if you need any further assistance.

Thanks,

Prashanth Are

MS Fabric community support

[v-prasare]

Hi @mortoman,

We would like to follow up to see if the solution provided by the community member resolved your issue. Please let us know if you need any further assistance.

@JMCK & @igm87 , thanks for your prompt response.

Thanks,

Prashanth Are

MS Fabric community support

If our community member response resolved your issue, please mark it as "Accept as solution" and click "Yes" if you found it helpful.

[JMCK]

I encountered the same error message, and to avoid any guesswork, I recommend the following steps:

1. Enable the Log Analytics Workspace for the Key Vault.
2. Activate the AuditEvent setting.
3. Attempt to enable the Customer Managed Key (CMK) at the workspace settings level.
4. Once these steps are completed, check the AzureDiagnostics in the Logs. Pay special attention to the "KeyUnwrap" operation name, as it was instrumental in resolving my issue.

Additionally, ensure that you have correctly granted access to the Object ID of the Microsoft Fabric CMK application, rather than its App ID. Initially, I mistakenly provided access to the App ID using Terraform, which led to the error.

This was the the error I found in my logs when I was facing the same issue:

Caller is not authorized to perform action on resource.
If role assignments, deny assignments or role definitions were changed recently, please observe propagation time.
Caller: appid=XXXX-XX-XXXX-XXXX-XXXXXX;oid=XXXX-XXX-XX-XX-XXXXXXXX;iss=https://sts.windows.net/XXXXXX-XX-XX-XXX-XXXXXXXX/
Action: 'Microsoft.KeyVault/vaults/keys/read'
Resource: '/subscriptions/AAA-BBBBB-CCCC-DDD-EEEEE/resourcegroups/rg-fabric-dev/providers/microsoft.keyvault/vaults/kv-fabric-dev/keys/cmkfabrictest'
Assignment: (not found)
DenyAssignmentId: null
DecisionReason: null
Vault: kv-fabric-dev;location=youlocation

[mortoman]

Hi JMCK,

To rule out human error I set everything up again, so my setup is,

New Key vault (purge protection and soft delete enabled) 

New RSA key

Recreated the Fabric Platform CMK app security principal with the Microsoft App ID

Assigned the role Key Vault Crypto Service Encryption User  to the security principal at both the key vault and the key (just to be sure)

Checked that the role is assigned to the correct object ID as you advised, which it is.

Checked Fabric tenant has CMK enabled

created a new empty workspace

added the key ID to the encryption settings for enabling CMK on the workspace

got the exact same error message.

I had enabled audit logging prior to all this as you suggested and got nothing in the logs relating to key issues!! I thought that was strange so I wrote a simple KQL to pull everything back to check logging was working and indeed everything comes back, but nothing related to the CMK key events.

Checked the logs again and again, but nothing related to key unwrap etc in there at all.

Its like its not even getting that far.

In desparation, I created a brand new Azure tenant and took out a new Fabric F2 capacity, I then did everything to the letter as per the Microsoft guide and it all worked first time (see below)!

I don't have control of the company platform so there must be something they have set to block this from completing. Thanks so much for all your advice, I'll keep going as I'm determined to find out what's stopping this now.

cheers 

[igm87]

hi,

I can confirm that you set the role at the KEY VAULT, you do not set it at KEY LEVEL. The process described in the guide works 100% as I have applied this myself by following the guide. The issue you have will be a conditional access policy preventing the process from completing. I would look at the CAP set by your admin as the message is not saying the setup is in error, it is saying the setup is correct but it cannot apply the CMK due to something else.

[mortoman]

Hi, the key is RSA 2048 and I did try your suggestion, but it did not fix the issue, still get the exact same error message.

To be honest, I'm not 100% convinced you would be setting this role at the key level. As per the guide; Fabric only supports versionless customer managed keys, from the guide... Fabric checks the key vault daily for a new version, and uses the latest version available... 

This would indicate to me that the role should be set at the key vault, otherwise as soon as a new key is present all access would stop until the role was manually assigned to the new key. In addition, the guide does explicitly state that the role should be set at the key vault, prior to the creation of the RSA key.

Thanks for your suggestion though, it was worth a try.

[JMCK]

Hey, you need to create the key RSA type and make sure you give the role "Key Vault Crypto Service Encryption User" at the Fabric Platform CMK but at KEY LEVEL IS NOT at KeyVault level.