'Message': 'Internal error MwcTokenValidationException.', 'Source': 15, 'ErrorCode': 0

[cesarniyo]

When I run my job manually everything looking good but when I trigger my pipeline to run the jobs
I am getting token issue between my pipeline and mlflow server (authentication issue )

any documentation on how to properly set this?:

Fatal Error during MLflow session: INTERNAL_ERROR: Response: {'Message': 'Internal error MwcTokenValidationException.', 'Source': 15, 'ErrorCode': 0}

Stopping Spark session...

Traceback (most recent call last):

  File "/mnt/var/hadoop/tmp/nm-secondary-local-dir/usercache/trusted-service-user/appcache/application_1768522818505_0001/container_1768522818505_0001_02_000001/train.py", line 125, in <module>

    raise e

  File "/mnt/var/hadoop/tmp/nm-secondary-local-dir/usercache/trusted-service-user/appcache/application_1768522818505_0001/container_1768522818505_0001_02_000001/train.py", line 89, in <module>

    mlflow.set_experiment(EXPERIMENT_NAME)

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/mlflow/tracking/fluent.py", line 143, in set_experiment

    experiment = client.get_experiment_by_name(experiment_name)

                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/mlflow/tracking/client.py", line 545, in get_experiment_by_name

    return self._tracking_client.get_experiment_by_name(name)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/mlflow/tracking/_tracking_service/client.py", line 233, in get_experiment_by_name

    return self.store.get_experiment_by_name(name)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/synapse/ml/mlflow/synapse_mlflow_utils.py", line 374, in wrapper

    res = function(*args, **kwargs)

          ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/synapse/ml/mlflow/synapse_mlflow_utils.py", line 426, in wrapper

    res = function(*args, **kwargs)

          ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/synapse/ml/mlflow/tracking_store.py", line 149, in get_experiment_by_name

    return super().get_experiment_by_name(experiment_name)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/synapse/ml/mlflow/synapse_mlflow_utils.py", line 374, in wrapper

    res = function(*args, **kwargs)

          ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py", line 323, in get_experiment_by_name

    response_proto = self._call_endpoint(GetExperimentByName, req_body)

                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/synapse/ml/mlflow/synapse_mlflow_utils.py", line 374, in wrapper

    res = function(*args, **kwargs)

          ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/mlflow/store/tracking/rest_store.py", line 60, in _call_endpoint

    return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/mlflow/utils/rest_utils.py", line 290, in call_endpoint

    response = verify_rest_response(response, endpoint)

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/synapse/ml/mlflow/tracking_store.py", line 52, in wrapper

    return origin_func(response, endpoint)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/trusted-service-user/cluster-env/trident_env/lib/python3.11/site-packages/mlflow/utils/rest_utils.py", line 173, in verify_rest_response

    raise RestException(json.loads(response.text))

mlflow.exceptions.RestException: INTERNAL_ERROR: Response: {'Message': 'Internal error MwcTokenValidationException.', 'Source': 15, 'ErrorCode': 0}

)

[cesarniyo]

@v-tsaipranay 
Thank you so much for your response -
would please elaborate more on what exactly considered to be the pipeline identity (sorry I am new to Fabric)

[v-tsaipranay]

Hi @cesarniyo ,

Thank you for reaching out to the Microsoft Fabric Community Forum. 

The MwcTokenValidationException is not caused by MLflow itself. Instead, it originates from the Microsoft Fabric/Synapse MLflow gateway when it cannot validate the identity token provided by the runtime. This is why running the job manually works, but running it through a pipeline causes it to fail. In an interactive Spark session, a valid workspace token is already present, but a pipeline uses a different identity (such as a service principal or managed identity) and does not automatically get a Fabric/Synapse token. When the pipeline initiates the Spark container and mlflow.set_experiment() calls the MLflow REST API, the gateway tries to validate the token, but if it’s missing or invalid, it raises MwcTokenValidationException. To fix this, make sure the pipeline’s identity has Contributor or higher access to the Fabric/Synapse workspace, or inject a valid workspace token into the environment before using MLflow. With a trusted identity, MLflow calls from the pipeline will work as they do in manual runs.

Thank you.