Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
DCHTan_SG
New Member

MLflow schema is only supported for scalar Spark types

Hi all, I was following the guide:

Create models with Automated ML (preview)

When I reached the step to train a LightGBMClassifier model on the provided training data, I encountered the following warning and errors. May I know if anyone has encountered this and any way to overcome this?

 

"WARNING mlflow.data.spark_dataset: Failed to infer schema for Spark dataset. Exception: Unsupported Spark Type '<class 'pyspark.ml.linalg.VectorUDT'>', MLflow schema is only supported for scalar Spark types."

 

Cell In[104], line 9

6 with mlflow.start_run(run_name="default") as run:

7 # Create a LightGBMClassifier model with specified settings

8 model = LightGBMClassifier(objective="binary", featuresCol="features", labelCol="Exited")

----> 9 model = model.fit(train_data)

 

File ~/cluster-env/trident_env/lib/python3.10/site-packages/mlflow/utils/autologging_utils/safety.py:573, in safe_patch.<locals>.safe_patch_function(*args, **kwargs)

571 patch_function.call(call_original, *args, **kwargs)

572 else:

--> 573 patch_function(call_original, *args, **kwargs)

575 session.state = "succeeded"

577 try_log_autologging_event(

578 AutologgingEventLogger.get_logger().log_patch_function_success,

579 session,

(...)

583 kwargs, 584 )

1 ACCEPTED SOLUTION

Hi @DCHTan_SG 

 

Apologies for the inconvenience.

Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

 

Thank you.

View solution in original post

3 REPLIES 3
v-cboorla-msft
Community Support
Community Support

Hi @DCHTan_SG 

 

Thanks for using Microsoft Fabric Community.

Autologging works by automatically capturing values of input parameters, output metrics, and output items of a machine learning model as it's being trained. This information is logged to your Microsoft Fabric workspace, where you can access and visualize it by using the MLflow APIs or the corresponding experiment and model items in your Microsoft Fabric workspace. This powerful feature allows you to log metrics, parameters, and models without the need for explicit log statements - all you need to do is call mlflow.autolog() before your training code. Auto-logging supports popular libraries such as Scikit-learn, XGBoost, PyTorch, Keras, Spark, and more.

For more details please refer : Autologging in Synapse Data Science - Microsoft Fabric | Microsoft Learn

Please refer this thread which might help you : How to do the Machine Model application in Fabric? - Microsoft Fabric Community 

If the issue still persists, please do let us know. Glad to help.

 

I hope this information helps. 

 

Thank you.

Thank you for the prompt feedback.

I already have mlflow.autolog(disable=True) .

I am not sure how autologging is related to this error:

"WARNING mlflow.data.spark_dataset: Failed to infer schema for Spark dataset. Exception: Unsupported Spark Type '<class 'pyspark.ml.linalg.VectorUDT'>', MLflow schema is only supported for scalar Spark types."

 

Maybe a goood way is for you to go through the "Create AutoML trials in Fabric - Microsoft Fabric | Microsoft Learn", "How-to guides", "Run AutoML trials".

Alternatively, is there a location where I can download this notebook?

 

Thank you!

Hi @DCHTan_SG 

 

Apologies for the inconvenience.

Please reach out to our support team to gain deeper insights and explore potential solutions. It's highly recommended that you reach out to our support team. Their expertise will be invaluable in suggesting the most appropriate approach.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

 

Thank you.

Helpful resources

Announcements
FabricCarousel_June2024

Fabric Monthly Update - June 2024

Check out the June 2024 Fabric update to learn about new features.

July Newsletter

Fabric Community Update - July 2024

Find out what's new and trending in the Fabric Community.

Top Kudoed Authors