Service Principals running specific Notebooks from Pipelines Fail. I believe I have isolated the issue to sempy.fabric.
Notes:
- This code runs perfectly in the notebook.
- It runs perfectly with no connection specified (Oath 2.0).
- Other notebooks in pipelines run fine with the same Service Principal with the same access to the same lakehouses/workspaces.
- Other notebooks using sempy.fabric also work, but not with this code. From another article, it's due to sempy.fabric using Oath 2.0 authentication internally (sometimes?).
- Edit: From above article, upgrading semantic-link by creating a new environment via an external repository in Fabric fixes this. The default version is 0.11. Please upgrade the default version in Fabric, Microsoft.
Here is the code:
import sempy.fabric as fabric
# Get the current workspace name
current_workspace_name = notebookutils.runtime.context["currentWorkspaceName"]
# List all workspaces
workspaces_df = fabric.list_workspaces()
# Determine the correct workspace ID
if current_workspace_name == "ws-prd-forge":
WorkspaceID = workspaces_df[workspaces_df['Name'] == "ws-prd-forge-data"]['Id'].values[0]
sql_aep = "{GUID}.datawarehouse.fabric.microsoft.com"
else:
WorkspaceID = workspaces_df[workspaces_df['Name'] == "ws-dev-forge-data"]['Id'].values[0]
sql_aep = "{GUID}.datawarehouse.fabric.microsoft.com"
# List all lakehouses in the selected workspace
lakehouses_list = notebookutils.lakehouse.list(WorkspaceID)
# Create variables for each lakehouse with "_path" appended
for lakehouse in lakehouses_list:
lakehouse_name = lakehouse["displayName"]
lakehouse_id = lakehouse["id"]
globals()[f"{lakehouse_name}_path"] = f"abfss://{WorkspaceID}@onelake.dfs.fabric.microsoft.com/{lakehouse_id}"
bronze_prefix = 'bronze_'
silver_prefix = 'silver_'
schema_prefix = '/Tables/dbo/'
Here is the error message:
Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - FabricHTTPException, Error value - 401 Unauthorized for url: https://api.powerbi.com/powerbi/globalservice/v201606/clusterdetails
Headers: {'Cache-Control': 'no-store, must-revalidate, no-cache', 'Pragma': 'no-cache', 'Transfer-Encoding': 'chunked', 'Content-Type': 'application/octet-stream', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'X-Frame-Options': 'deny', 'X-Content-Type-Options': 'nosniff', 'RequestId': 'b90defe2-ba4a-4922-b512-dcf218366fac', 'Access-Control-Expose-Headers': 'RequestId', 'Date': 'Tue, 05 May 2026 20:41:37 GMT'}' :
This code works as a replacement. This directly calls the REST API instead:
import requests
current_workspace_name = notebookutils.runtime.context['currentWorkspaceName']
# Replace fabric.list_workspaces() with direct API call
response = requests.get(
f'{base_url}/groups',
headers=headers
)
response.raise_for_status()
workspaces = response.json()['value']
# Determine the correct workspace ID
if current_workspace_name == 'ws-prd-forge':
target_name = 'ws-prd-forge-data'
sql_aep = '{GUID}.datawarehouse.fabric.microsoft.com'
else:
target_name = 'ws-dev-forge-data'
sql_aep = '{GUID}.datawarehouse.fabric.microsoft.com'
WorkspaceID = next(w['id'] for w in workspaces if w['name'] == target_name)
# Replace notebookutils.lakehouse.list() with direct API call
response = requests.get(
headers=headers
)
response.raise_for_status()
lakehouses_list = response.json()['value']
for lakehouse in lakehouses_list:
lakehouse_name = lakehouse['displayName']
lakehouse_id = lakehouse['id']
globals()[f'{lakehouse_name}_path'] = f'abfss://{WorkspaceID}@onelake.dfs.fabric.microsoft.com/{lakehouse_id}'
bronze_prefix = 'bronze_'
silver_prefix = 'silver_'
schema_prefix = '/Tables/dbo/'
