'Message': 'Internal error MwcTokenValidationException.', 'Source': 15, 'ErrorCode': 0

[cesarniyo]

When I run my job manually everything looking good but when I trigger my pipeline to run the jobs
I am getting token issue between my pipeline and mlflow server (authentication issue )

any documentation on how to properly set this?:

Fatal Error during MLflow session: INTERNAL_ERROR: Response: {'Message': 'Internal error MwcTokenValidationException.', 'Source': 15, 'ErrorCode': 0}

Stopping Spark session...

Traceback (most recent call last):

  File "/mnt/var/hadoop/tmp/nm-secondary-local-dir/usercache/trusted-service-user/appcache/application_1768522818505_0001/container_1768522818505_0001_02_000001/train.py", line 125, in <module>

    raise e

  File "/mnt/var/hadoop/tmp/nm-secondary-local-dir/usercache/trusted-service-user/appcache/application_1768522818505_0001/container_1768522818505_0001_02_000001/train.py", line 89, in <module>

    mlflow.set_experiment(EXPERIMENT_NAME)

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/mlflow/tracking/fluent.py", line 143, in set_experiment

    experiment = client.get_experiment_by_name(experiment_name)

                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/mlflow/tracking/client.py", line 545, in get_experiment_by_name

    return self._tracking_client.get_experiment_by_name(name)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/mlflow/tracking/_tracking_service/client.py", line 233, in get_experiment_by_name

    return self.store.get_experiment_by_name(name)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/synapse/ml/mlflow/synapse_mlflow_utils.py", line 374, in wrapper

    res = function(*args, **kwargs)

          ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/synapse/ml/mlflow/synapse_mlflow_utils.py", line 426, in wrapper

    res = function(*args, **kwargs)

          ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/synapse/ml/mlflow/tracking_store.py", line 149, in get_experiment_by_name

    return super().get_experiment_by_name(experiment_name)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/synapse/ml/mlflow/synapse_mlflow_utils.py", line 374, in wrapper

    res = function(*args, **kwargs)

          ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py", line 323, in get_experiment_by_name

    response_proto = self._call_endpoint(GetExperimentByName, req_body)

                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/synapse/ml/mlflow/synapse_mlflow_utils.py", line 374, in wrapper

    res = function(*args, **kwargs)

          ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py", line 60, in _call_endpoint

    return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/mlflow/utils/rest_utils.py", line 290, in call_endpoint

    response = verify_rest_response(response, endpoint)

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/synapse/ml/mlflow/tracking_store.py", line 52, in wrapper

    return origin_func(response, endpoint)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/mlflow/utils/rest_utils.py", line 173, in verify_rest_response

    raise RestException(json.loads(response.text))

mlflow.exceptions.RestException: INTERNAL_ERROR: Response: {'Message': 'Internal error MwcTokenValidationException.', 'Source': 15, 'ErrorCode': 0}

)

[v-tsaipranay]

Hi @cesarniyo ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions.

Thank you.

[v-tsaipranay]

Hi @cesarniyo ,

In Microsoft Fabric, when you run a job manually, it executes under your user identity, and Fabric automatically provides a valid workspace token. That’s why MLflow works in that case.

When the job is triggered via a pipeline, it runs under the pipeline execution identity (usually the workspace managed identity or a configured service principal), not your user account. This identity must be explicitly granted Contributor or higher access to the Fabric workspace. If it isn’t, the Fabric MLflow gateway cannot validate the token and throws MwcTokenValidationException.

To resolve your issue, please check the workspace access and ensure the pipeline’s execution identity has Contributor/Admin permissions. No code changes or manual token handling are required.

Once permissions are updated, MLflow calls from the pipeline should work the same way as manual runs.

Thank you.

[cesarniyo]

@v-tsaipranay 
Thank you so much for your response -
would please elaborate more on what exactly considered to be the pipeline identity (sorry I am new to Fabric)

[v-tsaipranay]

Hi @cesarniyo ,

Thank you for reaching out to the Microsoft Fabric Community Forum. 

The MwcTokenValidationException is not caused by MLflow itself. Instead, it originates from the Microsoft Fabric/Synapse MLflow gateway when it cannot validate the identity token provided by the runtime. This is why running the job manually works, but running it through a pipeline causes it to fail. In an interactive Spark session, a valid workspace token is already present, but a pipeline uses a different identity (such as a service principal or managed identity) and does not automatically get a Fabric/Synapse token. When the pipeline initiates the Spark container and mlflow.set_experiment() calls the MLflow REST API, the gateway tries to validate the token, but if it’s missing or invalid, it raises MwcTokenValidationException. To fix this, make sure the pipeline’s identity has Contributor or higher access to the Fabric/Synapse workspace, or inject a valid workspace token into the environment before using MLflow. With a trusted identity, MLflow calls from the pipeline will work as they do in manual runs.

Thank you.