Forum Discussion

macrohardy's avatar
macrohardy
Regular Visitor
4 months ago
Solved

Service Principals running specific Notebooks from Pipelines Fail

Service Principals running specific Notebooks from Pipelines Fail. I believe I have isolated the issue to sempy.fabric.

Notes:

  • This code runs perfectly in the notebook.
  • It runs perfectly with no connection specified (Oath 2.0).
  • Other notebooks in pipelines run fine with the same Service Principal with the same access to the same lakehouses/workspaces.
  • Other notebooks using sempy.fabric also work, but not with this code. From another article, it's due to sempy.fabric using Oath 2.0 authentication internally (sometimes?).
  • Edit: From above article, upgrading semantic-link by creating a new environment via an external repository in Fabric fixes this. The default version is 0.11. Please upgrade the default version in Fabric, Microsoft. 

    Here is the code:
import sempy.fabric as fabric

# Get the current workspace name
current_workspace_name = notebookutils.runtime.context["currentWorkspaceName"]

# List all workspaces
workspaces_df = fabric.list_workspaces()

# Determine the correct workspace ID
if current_workspace_name == "ws-prd-forge":
    WorkspaceID = workspaces_df[workspaces_df['Name'] == "ws-prd-forge-data"]['Id'].values[0]
    sql_aep = "{GUID}.datawarehouse.fabric.microsoft.com"
else:
    WorkspaceID = workspaces_df[workspaces_df['Name'] == "ws-dev-forge-data"]['Id'].values[0]
    sql_aep = "{GUID}.datawarehouse.fabric.microsoft.com"

# List all lakehouses in the selected workspace
lakehouses_list = notebookutils.lakehouse.list(WorkspaceID)

# Create variables for each lakehouse with "_path" appended
for lakehouse in lakehouses_list:
    lakehouse_name = lakehouse["displayName"]
    lakehouse_id = lakehouse["id"]
    globals()[f"{lakehouse_name}_path"] = f"abfss://{WorkspaceID}@onelake.dfs.fabric.microsoft.com/{lakehouse_id}"

bronze_prefix = 'bronze_'
silver_prefix = 'silver_'
schema_prefix = '/Tables/dbo/'
 

Here is the error message:
Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - FabricHTTPException, Error value - 401 Unauthorized for url: https://api.powerbi.com/powerbi/globalservice/v201606/clusterdetails
Headers: {'Cache-Control': 'no-store, must-revalidate, no-cache', 'Pragma': 'no-cache', 'Transfer-Encoding': 'chunked', 'Content-Type': 'application/octet-stream', 'Strict-Transport-Security': 'max-age=31536000; includeSubDomains', 'X-Frame-Options': 'deny', 'X-Content-Type-Options': 'nosniff', 'RequestId': 'b90defe2-ba4a-4922-b512-dcf218366fac', 'Access-Control-Expose-Headers': 'RequestId', 'Date': 'Tue, 05 May 2026 20:41:37 GMT'}' :

 

This code works as a replacement. This directly calls the REST API instead:

import requests

 

current_workspace_name = notebookutils.runtime.context['currentWorkspaceName']

 

# Replace fabric.list_workspaces() with direct API call
response = requests.get(
    f'{base_url}/groups',
    headers=headers
)
response.raise_for_status()
workspaces = response.json()['value']

 

# Determine the correct workspace ID
if current_workspace_name == 'ws-prd-forge':
    target_name = 'ws-prd-forge-data'
    sql_aep = '{GUID}.datawarehouse.fabric.microsoft.com'
else:
    target_name = 'ws-dev-forge-data'
    sql_aep = '{GUID}.datawarehouse.fabric.microsoft.com'

 

WorkspaceID = next(w['id'] for w in workspaces if w['name'] == target_name)

 

# Replace notebookutils.lakehouse.list() with direct API call
response = requests.get(
    headers=headers
)
response.raise_for_status()
lakehouses_list = response.json()['value']

 

for lakehouse in lakehouses_list:
    lakehouse_name = lakehouse['displayName']
    lakehouse_id = lakehouse['id']
    globals()[f'{lakehouse_name}_path'] = f'abfss://{WorkspaceID}@onelake.dfs.fabric.microsoft.com/{lakehouse_id}'

 

bronze_prefix = 'bronze_'
silver_prefix = 'silver_'
schema_prefix = '/Tables/dbo/'
security
  • v-sgandrathi's avatar
    v-sgandrathi
    4 months ago

    Hi macrohardy,

     

    This issue is likely due to the version of semantic-link (sempy.fabric) available in the default Fabric runtime.

    From the error and test results, it doesn't seem related to workspace permissions or Service Principal configuration, since:

    • the notebook runs interactively without issues
    • the same Service Principal works for other notebooks and lakehouse tasks
    • direct REST API calls are successful
    • the failure only happens with fabric.list_workspaces()

    The key detail is the internal call to:

    https://api.powerbi.com/powerbi/globalservice/v201606/clusterdetails

    Older semantic-link versions don't always handle Service Principal authentication correctly in Pipelines, especially with internal Power BI metadata APIs.

    Using direct REST API calls, as described above, is currently the most reliable workaround because it avoids the authentication flow used by sempy.fabric.

    A newer semantic-link version seems to fix this. Upgrading the package in a custom Fabric environment has been reported to resolve the problem.

    Right now, you can either:

    1. Use direct REST API calls as a workaround
    2. Create a custom Fabric environment with an updated semantic-link package
    3. Wait for Microsoft to update the default Fabric runtime package

    This appears to be a runtime or library limitation, not a notebook configuration issue.


    The following documents are provided for your reference:
    Enable service principal authentication for admin APIs - Microsoft Fabric | Microsoft Learn
    sempy.fabric package | Microsoft Learn
    Service Principal support for Semantic Link - Microsoft Fabric | Microsoft Learn

    Thank you.

3 Replies

  • v-sgandrathi's avatar
    v-sgandrathi
    Icon for Community Support rankCommunity Support
    4 months ago

    Hi macrohardy,

     

    This issue is likely due to the version of semantic-link (sempy.fabric) available in the default Fabric runtime.

    From the error and test results, it doesn't seem related to workspace permissions or Service Principal configuration, since:

    • the notebook runs interactively without issues
    • the same Service Principal works for other notebooks and lakehouse tasks
    • direct REST API calls are successful
    • the failure only happens with fabric.list_workspaces()

    The key detail is the internal call to:

    https://api.powerbi.com/powerbi/globalservice/v201606/clusterdetails

    Older semantic-link versions don't always handle Service Principal authentication correctly in Pipelines, especially with internal Power BI metadata APIs.

    Using direct REST API calls, as described above, is currently the most reliable workaround because it avoids the authentication flow used by sempy.fabric.

    A newer semantic-link version seems to fix this. Upgrading the package in a custom Fabric environment has been reported to resolve the problem.

    Right now, you can either:

    1. Use direct REST API calls as a workaround
    2. Create a custom Fabric environment with an updated semantic-link package
    3. Wait for Microsoft to update the default Fabric runtime package

    This appears to be a runtime or library limitation, not a notebook configuration issue.


    The following documents are provided for your reference:
    Enable service principal authentication for admin APIs - Microsoft Fabric | Microsoft Learn
    sempy.fabric package | Microsoft Learn
    Service Principal support for Semantic Link - Microsoft Fabric | Microsoft Learn

    Thank you.

  • v-sgandrathi's avatar
    v-sgandrathi
    Icon for Community Support rankCommunity Support
    4 months ago

    Hi macrohardy,

     

    We wanted to check if you had a chance to review our last reply. Let us know if it helped or if you need more guidance, we're always happy to help further.

    Looking forward to hearing from you!

     

    Thank you.

  • v-sgandrathi's avatar
    v-sgandrathi
    Icon for Community Support rankCommunity Support
    3 months ago

    Hi macrohardy,

     

    Following up to check whether you got a chance to review the suggestions given. If the issue still persists please let us know. Glad to help. 

     

    Thank you.