Private link performance degradation

[mwsmoley]

My company is starting to use Fabric Notebooks for various activities, and to secure our credentials, we implemented the usage of a Key Vault.  After setting this up and getting it to work, people noticed massive performance hits in starter pools and Python experience start-ups. For instance, the Python experience went from 5 to 10 seconds, starting to over 2 minutes on average and never below 1 minute.  During research, I was not able to find an explanation until I stumbled upon a note about spark sessions (see image below).  It seems that you have to decide between security and performance.  Does anyone have a suggestion on how to use Key Vault without private links?  Most of these scripts are going to be running in the background and Fabric is not a trusted resource so this was the only way I could get it to work.

Sincerly,

Michal

[V-yubandi-msft]

Hi @mwsmoley ,

Could you let us know if your issue has been resolved or if you need any additional information? If the issue is resolved, please mark it as the accepted solution. This helps other members who might be facing similar issues.

Thank You.

[mwsmoley]

Not really solved but I will mark it as accepted.

[V-yubandi-msft · Accepted solution]

Hi @mwsmoley ,

Thank you for your insights, @lbendlin . You are right  this involves not only a trade-off between security and performance but also cost. Keeping Spark pools  warm can become expensive.

To balance these factors, consider the following optimizations.
1. Reduce Key Vault Calls -  Cache secrets in memory or use environment variables to reduce authentication overhead.

2. Use Managed Identity -  Switching to Managed Identity can enhance security and performance while simplifying access management.

3. Optimize Spark Pool Usage -  Schedule workloads efficiently to minimize compute costs and improve resource allocation.

Hope this helps.

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.

[lbendlin]

 It seems that you have to decide between security and performance.

It's a bit more complex than that.  You have to decide between security, performance, and cost.  Keeping Spark pools "warm" is getting expensive really quickly.