Forum Discussion
Some methods in notebooks do not work when executed from Data Pipelines deployed via SPN
Hi, I already created the ticket with Microsoft (2504161420001430) and also opened the issue here and also seems to be related to this issue here. `notebookutils.runtime.context.get("currentW...
My team is also experiencing the same issue but when running notebooks called via API e.g. a making a request like this:
https://api.fabric.microsoft.com/v1/workspaces/<workspace_id>/items/<notebook_id>/jobs/instances?jobType=RunNotebook
from a service principal (in our case the managed identity of an ADF factory)
I imagine the Fabric Pipeline is doing the same API call under the hood.
As notebookutils.runtime.context.get("currentWorkspaceId", "") is working and we only have two workspaces at the moment (dev and prd) for a temporary workaround we have hardcoded environment specific values based on the workspace id into the custom package that we are using to manage ELT.
Still deciding whether or not we want to move these hardcoded environment specific values to a JSON file managed via CI/CD or swap back to notebookutils once the current issues are resolved. Would be nice to have less moving parts but also equally nice to have less reliance on MS managed bits and bobs that haven't proved completely reliable yet.
We have also observed the following log output when running a notebook via Service Princiapl that imports papermill (we have a development workflow where we can build out data modelling locally and run it via local spark installs before using CI/CD to push it up to Fabric, use papermill when running locally in place of notebookutils.notebook calls) this one is easier to tidy up just had to make sure we don't import papermill when running in a remote context but might help whoever is looking into this.
Failed to fetch cluster details Traceback (most recent call last): File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.11/site-packages/synapse/ml/fabric/service_discovery.py", line 110, in get_mlflow_shared_host raise Exception( Exception: Fetch cluster details returns 401:b'' Fetch cluster details returns 401:b'' Traceback (most recent call last): File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.11/site-packages/synapse/ml/fabric/service_discovery.py", line 152, in set_envs set_fabric_env_config(builder.fetch_fabric_client_param(with_tokens=False)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.11/site-packages/synapse/ml/fabric/service_discovery.py", line 72, in fetch_fabric_client_param shared_host = get_fabric_context().get("trident.aiskill.shared_host") or self.get_mlflow_shared_host(pbienv) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.11/site-packages/synapse/ml/fabric/service_discovery.py", line 110, in get_mlflow_shared_host raise Exception( Exception: Fetch cluster details returns 401:b'' ## Not In PBI Synapse Platform ##
from a service principal (in our case the managed identity of an ADF factory)
I imagine the Fabric Pipeline is doing the same API call under the hood.
As notebookutils.runtime.context.get("currentWorkspaceId", "") is working and we only have two workspaces at the moment (dev and prd) for a temporary workaround we have hardcoded environment specific values based on the workspace id into the custom package that we are using to manage ELT.
Still deciding whether or not we want to move these hardcoded environment specific values to a JSON file managed via CI/CD or swap back to notebookutils once the current issues are resolved. Would be nice to have less moving parts but also equally nice to have less reliance on MS managed bits and bobs that haven't proved completely reliable yet.
We have also observed the following log output when running a notebook via Service Princiapl that imports papermill (we have a development workflow where we can build out data modelling locally and run it via local spark installs before using CI/CD to push it up to Fabric, use papermill when running locally in place of notebookutils.notebook calls) this one is easier to tidy up just had to make sure we don't import papermill when running in a remote context but might help whoever is looking into this.
Failed to fetch cluster details Traceback (most recent call last): File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.11/site-packages/synapse/ml/fabric/service_discovery.py", line 110, in get_mlflow_shared_host raise Exception( Exception: Fetch cluster details returns 401:b'' Fetch cluster details returns 401:b'' Traceback (most recent call last): File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.11/site-packages/synapse/ml/fabric/service_discovery.py", line 152, in set_envs set_fabric_env_config(builder.fetch_fabric_client_param(with_tokens=False)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.11/site-packages/synapse/ml/fabric/service_discovery.py", line 72, in fetch_fabric_client_param shared_host = get_fabric_context().get("trident.aiskill.shared_host") or self.get_mlflow_shared_host(pbienv) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/trusted-service-user/cluster-env/clonedenv/lib/python3.11/site-packages/synapse/ml/fabric/service_discovery.py", line 110, in get_mlflow_shared_host raise Exception( Exception: Fetch cluster details returns 401:b'' ## Not In PBI Synapse Platform ##