Forum Discussion

mmilosanovic's avatar
mmilosanovic
Advocate IV
1 year ago

Some methods in notebooks do not work when executed from Data Pipelines deployed via SPN

Hi,   I already created the ticket with Microsoft (2504161420001430) and also opened the issue here and also seems to be related to this issue here.   `notebookutils.runtime.context.get("currentW...
Data Engineering
Data Pipeline
Notebook
annhwallinger's avatar
annhwallinger
Helper I
1 year ago

Microsoft responded to my case yesterday indicating that synapsesql() does not yet support SPN and there is no deployment date for when it will. 

    • g3kuser's avatar
      g3kuser
      Helper II
      1 year ago

      My ingestion process, which uses pipelines and notebooks, is deployed through the Fabric CI/CD library under SPN ownership. I have extensively used notebookutils (such as GetToken, Mount, getSecret, getWithProperties, updateDefinition, and env methods), sempy (using FabricRestClient to read the list of workspaces, lakehouses, and create shortcuts), and MSAL libraries.

      Until last week, I was encountering cluster errors related to the sempy library and the execution of the spark.sql method under the SPN. However, these issues have disappeared as of this week. The process has been running successfully under the SPN for the past four days.

      The SPN I am using is not a workspace identity, but a separate SPN created in our tenant with Contributor permissions on Fabric workspaces and read-only Fabric API access.

      I have an ongoing support request for the issues I encountered under the SPN. Based on diagnostic logs, I was initially informed it was a permission issue. However, something seems to have changed behind the scenes this week—despite no changes made to permissions on my end, recent retests show successful execution.


      Thanks,

      Gayatri

      • annhwallinger's avatar
        annhwallinger
        Helper I
        1 year ago

        Thank you for the info. That all sounds pretty much same as my experience except that we use the synapsesql() function and that still does not work and is not supporting SPN per Microsoft so it all fails at that step and cannot move forward. The other functions before it gets to that one that used to fail are working now.