Forum Discussion

going_grey's avatar
going_grey
Frequent Visitor
5 months ago
Solved

Streaming Spark Job - KQL Authentication

Hi   I want to stream data into a KQL database using a Streaming Spark Job. My initial plan was to retrieve a service principal secret from a key vault using:   notebookutils.credentials.getSecr...
one lake
Real-Time hub
  • v-veshwara-msft's avatar
    v-veshwara-msft
    4 months ago

    Hi going_grey ,

     

    Thanks for pointing that out, you’re correct.

    Even with the SDK approach, a credential is still required to authenticate to Key Vault, so it does not remove the need to securely store and pass a secret.

     

    Because of this, there seems to be isn’t a fully secure or secretless pattern available today for this scenario in Spark streaming with KQL, and this is a current limitation rather than a configuration issue.

     

    For production scenarios, approaches where authentication is handled outside Spark (for example using Fabric Data Pipeline or Eventstream) tend to be more reliable with the current capabilities.

     

    Hope this helps. Please reach out for further assistance.
    Thank you.

deborshi_nag's avatar
deborshi_nag
Super User
5 months ago

Hello going_grey 

 

You’re correct that notebookutils.credentials.getSecret(vault_url, secret_name) is the supported way to read a secret from Azure Key Vault in a Fabric notebook. However, it’s important to separate Key Vault access from KQL authentication support.

 

When you call getSecret, the identity used depends on how the notebook is executed:

  • Running the notebook interactively uses your user identity
  • Running it via a pipeline uses the pipeline execution identity, which may be a service principal if configured

That identity must have Key Vault Secrets User permissions on the vault.

 

However, even if you successfully retrieve the service principal secret, this does not currently unblock streaming into a KQL database. At present:

  • AKV References are not supported in Spark / notebooks
  • Workspace Managed Identity is not supported for KQL authentication
  • Spark streaming jobs therefore cannot authenticate to KQL using Fabric-managed identities

As a result, using Key Vault to retrieve a service principal secret from a Spark streaming job does not provide a fully supported authentication path into KQL today. Until KQL supports Workspace Managed Identity or Fabric-native identities for Spark ingestion, this remains a platform limitation rather than a Key Vault issue.