topic Re: Private link performance degradation in Data Engineering

Private link performance degradation

mwsmoley — Thu, 27 Feb 2025 13:57:54 GMT

My company is starting to use Fabric Notebooks for various activities, and to secure our credentials, we implemented the usage of a Key Vault. After setting this up and getting it to work, people noticed massive performance hits in starter pools and Python experience start-ups. For instance, the Python experience went from 5 to 10 seconds, starting to over 2 minutes on average and never below 1 minute. During research, I was not able to find an explanation until I stumbled upon a note about spark sessions (see image below). It seems that you have to decide between security and performance. Does anyone have a suggestion on how to use Key Vault without private links? Most of these scripts are going to be running in the background and Fabric is not a trusted resource so this was the only way I could get it to work.

Sincerly,

Michal

Re: Private link performance degradation

lbendlin — Fri, 28 Feb 2025 00:29:17 GMT

It seems that you have to decide between security and performance.

It's a bit more complex than that. You have to decide between security, performance, and cost. Keeping Spark pools "warm" is getting expensive really quickly.

Re: Private link performance degradation

V-yubandi-msft — Fri, 28 Feb 2025 06:49:10 GMT

Hi mwsmoley ,

Thank you for your insights, lbendlin . You are right this involves not only a trade-off between security and performance but also cost. Keeping Spark pools warm can become expensive.

To balance these factors, consider the following optimizations.
1. Reduce Key Vault Calls - Cache secrets in memory or use environment variables to reduce authentication overhead.

2. Use Managed Identity - Switching to Managed Identity can enhance security and performance while simplifying access management.

3. Optimize Spark Pool Usage - Schedule workloads efficiently to minimize compute costs and improve resource allocation.

Hope this helps.

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Private link performance degradation

V-yubandi-msft — Wed, 05 Mar 2025 10:07:23 GMT

Hi mwsmoley ,

Could you let us know if your issue has been resolved or if you need any additional information? If the issue is resolved, please mark it as the accepted solution. This helps other members who might be facing similar issues.

Thank You.

Re: Private link performance degradation

mwsmoley — Wed, 05 Mar 2025 12:58:23 GMT

Not really solved but I will mark it as accepted.