Forum Discussion
Streaming Spark Job - KQL Authentication
Hi going_grey ,
Thanks for pointing that out, you’re correct.
Even with the SDK approach, a credential is still required to authenticate to Key Vault, so it does not remove the need to securely store and pass a secret.
Because of this, there seems to be isn’t a fully secure or secretless pattern available today for this scenario in Spark streaming with KQL, and this is a current limitation rather than a configuration issue.
For production scenarios, approaches where authentication is handled outside Spark (for example using Fabric Data Pipeline or Eventstream) tend to be more reliable with the current capabilities.
Hope this helps. Please reach out for further assistance.
Thank you.
Hi going_grey ,
Thanks for reaching out to Microsoft Fabric Community.
From your scenario, using a Service Principal for KQL authentication is valid. The limitation is in Spark execution, where streaming or scheduled jobs do not have an interactive user context and currently do not support Service Principal or workspace identity for secret retrieval or KQL authentication. As a result, even if the secret is stored correctly in Key Vault, it cannot be used in a fully supported way during Spark streaming execution.
As a workaround, you can authenticate to Key Vault programmatically using the Azure SDK (for example via azure-identity and Key Vault client libraries) and retrieve the secret using a Service Principal instead of relying on notebookutils.credentials.getSecret(). This works in Spark jobs, but you will still need to securely pass and manage the client secret.
Reference: Access Azure Key Vault secrets from MS Fabric Notebooks
For production scenarios, you may want to consider handling authentication and ingestion outside Spark, such as using Fabric Data Pipeline or Eventstream.
For reference:
NotebookUtils (former MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn
Also, you can explore Fabric Connections in notebooks to define connections using Service Principal or workspace identity.
Hope this helps. Please reach out for further assistance.
Thank you.
v-veshwara-msft wrote:As a workaround, you can authenticate to Key Vault programmatically using the Azure SDK (for example via azure-identity and Key Vault client libraries) and retrieve the secret using a Service Principal instead of relying on notebookutils.credentials.getSecret(). This works in Spark jobs, but you will still need to securely pass and manage the client secret.
If you use the Azure SDK with a service principal to retrieve "the secret you actually need", then you'd still need to provide a secret to establish the key vault connection (which also needs to be secured).
Hi going_grey ,
Thanks for pointing that out, you’re correct.
Even with the SDK approach, a credential is still required to authenticate to Key Vault, so it does not remove the need to securely store and pass a secret.
Because of this, there seems to be isn’t a fully secure or secretless pattern available today for this scenario in Spark streaming with KQL, and this is a current limitation rather than a configuration issue.
For production scenarios, approaches where authentication is handled outside Spark (for example using Fabric Data Pipeline or Eventstream) tend to be more reliable with the current capabilities.
Hope this helps. Please reach out for further assistance.
Thank you.