Forum Discussion
'Message': 'Internal error MwcTokenValidationException.', 'Source': 15, 'ErrorCode': 0
When I run my job manually everything looking good but when I trigger my pipeline to run the jobs
I am getting token issue between my pipeline and mlflow server (authentication issue )
any documentation on how to properly set this?:
Hi cesarniyo ,
In Microsoft Fabric, when you run a job manually, it executes under your user identity, and Fabric automatically provides a valid workspace token. That’s why MLflow works in that case.
When the job is triggered via a pipeline, it runs under the pipeline execution identity (usually the workspace managed identity or a configured service principal), not your user account. This identity must be explicitly granted Contributor or higher access to the Fabric workspace. If it isn’t, the Fabric MLflow gateway cannot validate the token and throws MwcTokenValidationException.
To resolve your issue, please check the workspace access and ensure the pipeline’s execution identity has Contributor/Admin permissions. No code changes or manual token handling are required.
Once permissions are updated, MLflow calls from the pipeline should work the same way as manual runs.
Thank you.
8 Replies
- binitafulpagare
Kudo Commander
Hi cesarniyo,
Thanks for sharing the detailed error message.
Based on the traceback, it looks like the issue is related to MLflow authentication or token validation when the Spark job is executed through the Fabric pipeline.
Since the job runs successfully when triggered manually but fails during pipeline execution, one possible reason is a difference in the execution identity or authentication context between interactive notebook execution and pipeline-triggered execution.
I would suggest checking the following:
1. **Execution Identity:** Verify which identity is used when the pipeline triggers the Spark job.
2. **MLflow Authentication:** Check whether the pipeline execution context has the required permissions to access the MLflow tracking service.
3. **Workspace Permissions:** Ensure the executing identity has appropriate access to the workspace and MLflow experiment.
4. **Token Handling:** Avoid relying on interactive session tokens. Check whether authentication is correctly established in the pipeline execution context.
5. **Runtime Compatibility:** Verify that the MLflow and Synapse ML library versions are compatible with your Fabric Spark environment.
One important thing to investigate is whether the pipeline-triggered execution uses the same identity and permissions as the manual execution.
It would also be helpful to know whether you are triggering a Fabric notebook or a Spark job definition through the pipeline.
Hope these checks help narrow down the issue!
Best regards,
Binita Fulpagare
v-achippaCommunity Support
Hi Georg_Kal,
Thank you for reaching out to Microsoft Fabric Community.
Since this is an old thread which is resolved, I request you to please raise a new thread in the fabric community with the exact issue and the error details so that we can look into the issue properly and provide you the best solution possible.
Thanks and regards,
Anjan Kumar ChippaHi there,
I am facing the same issue. The service principal has admin rights but we still get the same error code. Do you have any other recommendations to fix this error?
Thanks,Georg
v-saisrao-msftHI cesarniyo,
Checking in to see if your issue has been resolved. let us know if you still need any assistance.
Thank you.
Hi cesarniyo ,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.
Thank you.
Hi cesarniyo ,
In Microsoft Fabric, when you run a job manually, it executes under your user identity, and Fabric automatically provides a valid workspace token. That’s why MLflow works in that case.
When the job is triggered via a pipeline, it runs under the pipeline execution identity (usually the workspace managed identity or a configured service principal), not your user account. This identity must be explicitly granted Contributor or higher access to the Fabric workspace. If it isn’t, the Fabric MLflow gateway cannot validate the token and throws MwcTokenValidationException.
To resolve your issue, please check the workspace access and ensure the pipeline’s execution identity has Contributor/Admin permissions. No code changes or manual token handling are required.
Once permissions are updated, MLflow calls from the pipeline should work the same way as manual runs.
Thank you.
Anonymous
Thank you so much for your response -
would please elaborate more on what exactly considered to be the pipeline identity (sorry I am new to Fabric)Hi cesarniyo ,
Thank you for reaching out to the Microsoft Fabric Community Forum.
The MwcTokenValidationException is not caused by MLflow itself. Instead, it originates from the Microsoft Fabric/Synapse MLflow gateway when it cannot validate the identity token provided by the runtime. This is why running the job manually works, but running it through a pipeline causes it to fail. In an interactive Spark session, a valid workspace token is already present, but a pipeline uses a different identity (such as a service principal or managed identity) and does not automatically get a Fabric/Synapse token. When the pipeline initiates the Spark container and mlflow.set_experiment() calls the MLflow REST API, the gateway tries to validate the token, but if it’s missing or invalid, it raises MwcTokenValidationException. To fix this, make sure the pipeline’s identity has Contributor or higher access to the Fabric/Synapse workspace, or inject a valid workspace token into the environment before using MLflow. With a trusted identity, MLflow calls from the pipeline will work as they do in manual runs.
Thank you.
