Forum Discussion
401/403 on sempy + Power BI REST API refresh endpoint when notebook is triggered via Service
Hello,
I'm running a PySpark notebook in Microsoft Fabric that does two things:
- Syncs a semantic model's TMSL definition against a warehouse schema using the Fabric updateDefinition API
- Triggers a full refresh via https://api.powerbi.com/v1.0/myorg/groups/{workspace_id}/datasets/{semantic_model_id}/refreshes
The notebook is executed through a Data Factory pipeline, and the pipeline is triggered using a Service Principal. That SP is the Submitter of the notebook, it has Admin role on the workspace, and the tenant setting "Service principals can use Fabric APIs" is enabled.
When the SP is the submitter, I consistently get 401 or 403 errors on two points:
- fabric.resolve_workspace_id() from sempy.fabric: this internally calls Power BI endpoints (/groups) using a token that appears to be scoped for Fabric (https://api.fabric.microsoft.com/.default), not Power BI. The scope mismatch causes the 401/403.
- https://api.powerbi.com/v1.0/myorg/... refresh endpoint: same problem: the token acquired via mssparkutils.credentials.getToken("https://analysis.windows.net/powerbi/api") doesn't seem to work correctly under the SP submitter context, even though it appears to return a token without error.
I've read this post that explains the scope mismatch issue with semantic-link and suggests upgrading to >= 0.12. That may help the resolve_workspace_id() call, but the core question I still have is:
- Can a notebook running under a Service Principal submitter successfully call api.powerbi.com refresh endpoints at all?
My understanding is that Power BI REST APIs for dataset refresh (/v1.0/myorg/groups/...) require a delegated user context for some operations and therefore service principals may be blocked on specific endpoints regardless of workspace role. But I haven't found a definitive answer.
My specific questions:
- Has anyone gotten api.powerbi.com/v1.0/myorg/groups/{id}/datasets/{id}/refreshes to work when the notebook submitter is a service principal? If so, what token acquisition approach did you use inside the notebook?
- Is mssparkutils.credentials.getToken("https://analysis.windows.net/powerbi/api") reliable when the submitter is an SP, or does it silently return an invalid/scoped-down token?
- As an alternative, are there other ways to obtain the workspace ID without using a parameter? Also, should the enhanced refresh be replaced with a semantic model refresh activity in the pipeline?
Any help or confirmation from others who've hit this wall would be hugely appreciated!
Thanks!
5 Replies
Hi jbauti10 ,
Thank you for sharing the details. The behavior might be related to the token scope used when the notebook runs under a Service Principal, since Power BI REST APIs require a token scoped for
https://analysis.windows.net/powerbi/api. If the notebook retrieves a Fabric scoped token instead, it may result in 401/403 errors.
Also, you mentioned a reference link in your post. Unfortunately, I don’t seem to have access to that content from my side. If possible, could you please share the relevant details or a publicly accessible reference so we can review it further.
In the meantime, these Microsoft docs might be helpful.
Datasets - Refresh Dataset In Group - REST API (Power BI Power BI REST APIs) | Microsoft LearnIf the issue persists, please let us know.
Hello V-yubandi-msft
Thank you for your reply.
I am indeed already using a token scoped for the Power BI API:
token = mssparkutils.credentials.getToken("https://analysis.windows.net/powerbi/api") headers = { "Authorization": f"Bearer {token}", "Content-Type": "application/json", }This token is used for both Fabric REST API and Power BI REST API calls, but the 401/403 only occurs on the Power BI REST API endpoints. The Fabric REST API calls succeed with the same token.
Regarding the reference link: it explains that older versions of semantic-link route workspace resolution through Power BI endpoints (/groups) while using a Fabric-scoped token internally, causing the mismatch. Upgrading to semantic-link >= 0.12 resolves that specific issue.
I was able to work around the problem by:
- Passing the workspace ID directly as a pipeline parameter using @pipeline().DataFactory instead of resolving it via sempy.
- Removing the refresh call from the notebook entirely and replacing it with a Semantic Model Refresh Activity in the pipeline after the schema sync step taken place in the notebook.
Both workarounds avoid the problematic Power BI REST API call from within the notebook context.
That said, I'm still curious why a token explicitly scoped to https://analysis.windows.net/powerbi/api — acquired via mssparkutils.credentials.getToken — fails on Power BI endpoints when the notebook submitter is a Service Principal. Any insights on why this is happening would be very helpful.
Thanks!
Hi jbauti10 ,
Thank you for sharing the update. Service principals are able to call the Power BI dataset refresh API if the tenant setting Allow service principals to use Power BI APIs is enabled and the service principal has access to the workspace.
However, when notebooks are run from a pipeline using a Service Principal submitter, the token from mssparkutils.credentials.getToken() may act differently since it’s generated within the Fabric runtime context. This can sometimes cause 401/403 errors on Power BI REST API endpoints, even though Fabric API calls work.
Your solution of passing the workspace ID from the pipeline and using the Semantic Model Refresh activity is a solid workaround that avoids this issue.
Regards,
Yugandhar.
Hi jbauti10 ,
Could you let us know if your issue has been resolved or if you are still experiencing difficulties? Your feedback is valuable to the community and can help others facing similar problems.
Hi jbauti10 ,
We haven’t received a response from your end yet. Please let us know whether the issue has been resolved or if you’re still facing any difficulties. Feel free to reach out if you need further assistance.
Thank you.
