Forum Discussion

DCHTan_SG's avatar
DCHTan_SG
New Member
2 years ago
Solved

MLflow schema is only supported for scalar Spark types

Hi all, I was following the guide: Create models with Automated ML (preview) When I reached the step to train a LightGBMClassifier model on the provided training data, I encountered the following w...
ml model
  • v-cboorla-msft's avatar
    v-cboorla-msft
    2 years ago

    Hi DCHTan_SG 

     

    Apologies for the inconvenience.

    Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

    Please go ahead and raise a support ticket to reach our support team:

    https://support.fabric.microsoft.com/support

    After creating a Support ticket please provide the ticket number as it would help us to track for more information.

     

    Thank you.

v-cboorla-msft's avatar
v-cboorla-msft
Icon for Microsoft Employee rankMicrosoft Employee
2 years ago

Hi DCHTan_SG 

 

Thanks for using Microsoft Fabric Community.

Autologging works by automatically capturing values of input parameters, output metrics, and output items of a machine learning model as it's being trained. This information is logged to your Microsoft Fabric workspace, where you can access and visualize it by using the MLflow APIs or the corresponding experiment and model items in your Microsoft Fabric workspace. This powerful feature allows you to log metrics, parameters, and models without the need for explicit log statements - all you need to do is call mlflow.autolog() before your training code. Auto-logging supports popular libraries such as Scikit-learn, XGBoost, PyTorch, Keras, Spark, and more.

For more details please refer : Autologging in Synapse Data Science - Microsoft Fabric | Microsoft Learn

Please refer this thread which might help you : How to do the Machine Model application in Fabric? - Microsoft Fabric Community 

If the issue still persists, please do let us know. Glad to help.

 

I hope this information helps. 

 

Thank you.

  • DCHTan_SG's avatar
    DCHTan_SG
    New Member
    2 years ago

    Thank you for the prompt feedback.

    I already have mlflow.autolog(disable=True) .

    I am not sure how autologging is related to this error:

    "WARNING mlflow.data.spark_dataset: Failed to infer schema for Spark dataset. Exception: Unsupported Spark Type '<class 'pyspark.ml.linalg.VectorUDT'>', MLflow schema is only supported for scalar Spark types."

     

    Maybe a goood way is for you to go through the "Create AutoML trials in Fabric - Microsoft Fabric | Microsoft Learn", "How-to guides", "Run AutoML trials".

    Alternatively, is there a location where I can download this notebook?

     

    Thank you!

    • v-cboorla-msft's avatar
      v-cboorla-msft
      Icon for Microsoft Employee rankMicrosoft Employee
      2 years ago

      Hi DCHTan_SG 

       

      Apologies for the inconvenience.

      Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

      Please go ahead and raise a support ticket to reach our support team:

      https://support.fabric.microsoft.com/support

      After creating a Support ticket please provide the ticket number as it would help us to track for more information.

       

      Thank you.