Sparkmonitor and secure strings

[morz3d]

Hi all,
I recently created pipeline where i send credentials stored as secure string from pipeline to notebook for further authentication.

When pipeline is running input paraemters are masked however what is exactly what i wanted however I was surprised when i open Sparkmonitor and in items snapshot i found all secure strings completly exposed. Is there a way not to display parameters here or to mask them?

[Anonymous · Accepted solution]

Hi @morz3d 

I can reproduce this, but not sure if this is by design. According to the following two old threads from six months ago, the behavior is different now. Back then, the SecureString type parameters passed from the pipeline were masked with asterisks in the SparkMonitor for the notebook activity. The actual values retrieved were also asterisks instead of the real values, causing the actual value transmission to fail. That looks like an issue. 

Passing SecureString Parameter Values in notebooks - Microsoft Fabric Community

pyspark - Passing SecureString Parameter Values in Fabric Notebooks - Stack Overflow

Now the behavior has changed. The SecureString type parameters pass the actual values but they are not masked in Sparkmonitor. This result does not seem secure enough. I haven't found documentation on what the scope of this SecureString works by now.

As a possible way to secure the credentials, you can use Azure Key Vault to store the secrets. Azure Key Vault is a cloud service for securely storing and accessing secrets. A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, or cryptographic keys. In the data pipeline, create a Web activity to get the secret from Azure Key Vault and set the Secure Output option to true to prevent the secret value from being logged in plain text. Then use it in the notebook activity. This might be helpful. 

This is a blog that might be helpful:

How-To Access Azure Key Vault Secrets from Fabric Notebook - Syntera

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!