topic Re: Cannot convert numpy type object to spark type in Data Science

Cannot convert numpy type object to spark type

Anonymous — Mon, 23 Dec 2024 14:13:47 GMT

I am new to MS fabric and trying to do a tutorial hands on - https://learn.microsoft.com/en-us/fabric/data-science/tutorial-data-science-batch-scoring

I am running the following code to Instantiate MLFlowTransformer object

from synapse.ml.predict import MLFlowTransformer

model = MLFlowTransformer( inputCols=list(df_test.columns),

outputCol='predictions',

modelName='lgbm_sm',

modelVersion=1 )

It is failing with following error

ValueError: Cannot convert numpy type object to spark type

How do I resolve this error? Is it something to do with list(df_test.columns)? Please advice.

Re: Cannot convert numpy type object to spark type

Anonymous — Tue, 24 Dec 2024 05:38:33 GMT

Hi Anonymous

I followed the tutorials without experiencing any error. Instead of entering codes manually, I downloaded and imported notebooks from microsoft/fabric-samples · GitHub. You can try running these notebooks and see if they work. Ensure that you need to attach the same lakehouse you used in the other parts of this series.

For the error you encountered, you can check the output of the previous code cell of the problem code. At the same time make sure that the attached lakehouse is correct. "df_test" is a table created in Tutorial Part 3.

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!

Re: Cannot convert numpy type object to spark type

Anonymous — Tue, 24 Dec 2024 10:55:29 GMT

Thank you for taking time to reply to my question. I followed the same steps exactly as you mentioned in your reply. Downloaded the notebooks from GitHub, followed the sequence of steps as mentioned in tutorial, no changes to note book code, all steps ran absolutely fine until this particular cell as mentioned in my initial question. Any more insights as to what is causing this error and how to resolve it?

Re: Cannot convert numpy type object to spark type

Baludesineti — Wed, 25 Dec 2024 17:53:05 GMT

yes, me also getting same error. can any one please tell us. how to slove it

Re: Cannot convert numpy type object to spark type

Anonymous — Fri, 27 Dec 2024 03:22:56 GMT

Hi Anonymous

I tested it again and it still worked at my end. I got nothing but a warning.

I guess maybe our environment is a little different? I ran the notebook in Runtime 1.2. My service version is 13.0.24766.49 and data region is UK South (London). Can you check your environment?

Best Regards,
Jing

Re: Cannot convert numpy type object to spark type

DrewAnderson — Mon, 10 Feb 2025 06:15:30 GMT

It seems like the issue might be related to the format of the DataFrame you're using. Make sure you're working with a valid Spark DataFrame, not a pandas one, as Spark expects a different structure.

Also, double-check that your model name and version are correct and compatible with your setup. It’s a good idea to ensure that your Spark and MLFlow versions are aligned with MS Fabric as well.

Re: Cannot convert numpy type object to spark type

Anonymous — Wed, 26 Feb 2025 22:04:03 GMT

Hi Anonymous
Thank you for your answers, I followed your suggestion and switched to Runtime 1.2. I can ensure that I did upload and execute those notebooks without any modification, and I tried to run all the previous 3 notebooks and used the same Runtime 1.2.
Unfortunately, I'm facing this new error

any idea how to fix this? Thanks.

Re: Cannot convert numpy type object to spark type

MiSchroe — Thu, 06 Mar 2025 10:35:46 GMT

I ran into the same problem.

BTW, I used the notebook from Explore a sample in Fabric.

It turned out that the "Exited" column was converted into a string where the plot over the categorical columns was coded. Instead, the Exited column has to be of type integer before you feed it into the training.

You can check in your experiment in the MLmodel file:

In line 34 the dtype should be int64, not object!

Don't forget to adapt the version number to your latest version of the model in the call to MLFlowTransformer.

Hope it helps!

Michael

Re: Cannot convert numpy type object to spark type

jchAgAr — Tue, 01 Apr 2025 13:49:41 GMT

MiSchroe - Thanks for clarifying the cause of the error. Pardon my ignorance, but could you explain how you would change the dtype from object to int64 in the context of this data science tutorial. Any help would be greatly appreciated as I'd like to better understand how this all fits together and complete it.

Re: Cannot convert numpy type object to spark type

MiSchroe — Wed, 02 Apr 2025 09:26:37 GMT

In the notebook 3-train-evaluate I have changed the data type of the Exited column to int:

NOTE: In row 3 I load the table dbo.df_clean, because I have used a Lakehouse with the new schema. In the original tutorial the schema has to be omitted.

Re: Cannot convert numpy type object to spark type

fikakagyta — Sat, 10 May 2025 17:56:40 GMT

The error you're seeing —
ValueError: Cannot convert numpy type object to spark type — typically means that Spark is encountering a data type in your input that it doesn't know how to convert into its internal types, and it often relates to using numpy.object_ or dtype=object in your DataFrame columns.

Root Cause:

In your code:

model = MLFlowTransformer(
    inputCols=list(df_test.columns),
    outputCol='predictions',
    modelName='lgbm_sm',
    modelVersion=1
)

You're passing list(df_test.columns) as the inputCols. This itself is likely not the problem.

The real issue is most likely with the df_test DataFrame — it seems to be a Pandas DataFrame (or a Spark DataFrame with some columns of ambiguous/numpy object type), and SynapseML expects a Spark DataFrame with well-defined schema and no ambiguous types.

✅ How to Fix It:

Ensure df_test is a Spark DataFrame, not a Pandas one. If it's currently a Pandas DataFrame, convert it:

spark_df_test = spark.createDataFrame(df_test)

But even when converting, Spark sometimes can't infer schema properly from object dtype. So it’s safer to:

Make sure df_test has clear types before conversion:

import pandas as pd

# Ensure all columns have concrete types (avoid object dtype)
df_test = df_test.astype({
    'col1': 'float64',
    'col2': 'int64',
    'col3': 'string',  # Replace with your actual column names/types
    # ...
})

Then convert:

spark_df_test = spark.createDataFrame(df_test)

Pass the Spark DataFrame to your pipeline:

model = MLFlowTransformer(
    inputCols=spark_df_test.columns,
    outputCol='predictions',
    modelName='lgbm_sm',
    modelVersion=1
)

transformed = model.transform(spark_df_test)

✅ Summary

Ensure you're using a Spark DataFrame, not a Pandas one.
Make sure all columns in your DataFrame have explicit types, especially no object types.
Use .astype() in Pandas before converting to Spark DataFrame.
Use spark.createDataFrame() for conversion.

Would you like help inferring the correct schema for your DataFrame or code to automatically clean the column types?

Re: Cannot convert numpy type object to spark type

Ali_Saghi — Mon, 22 Dec 2025 13:42:34 GMT

Great point. It solved my problem!
At first I went to convert all booleans to int, and of course it did not work. The problem was "Exited," as you mentioned already.

Re: Cannot convert numpy type object to spark type

deborshi_nag — Tue, 23 Dec 2025 12:19:14 GMT

Hello

This error usually comes from a type mismatch between what MLFlowTransformer (and the underlying pandas UDF it uses) expects and what your DataFrame is actually providing.

ValueError: Cannot convert numpy type object to spark type typically means that somewhere in the pipeline Spark is seeing a NumPy object dtype (from pandas or an Arrow conversion) and doesn’t know how to map it to a Spark type. This often happens when:

You pass a pandas DataFrame (or columns derived from pandas with object dtype) instead of a Spark DataFrame.
Your Spark columns contain arrays, maps, or mixed-type columns that Spark inferred as object (via Arrow/pandas), or they’re null-heavy and Spark guessed an ambiguous type.
The MLflow model signature expects certain columns/types (e.g., a single features vector) but you’re passing raw columns as inputCols that include non-primitive types or types that don’t match the model signature.

# If you accidentally used pandas:

import pandas as pd

from pyspark.sql.types import *

# Suppose you have a pandas DataFrame `pdf_test`

# Clean up pandas object dtypes to concrete types first

for col in pdf_test.columns:

if pdf_test[col].dtype == 'object':

# If it's actually numeric but stored as object

try:

pdf_test[col] = pd.to_numeric(pdf_test[col])

except Exception:

# Otherwise keep as string

pdf_test[col] = pdf_test[col].astype(str)

# Then define a schema (recommended)

schema = StructType([

# Adjust types to match your data and your model signature

StructField("col1", DoubleType()),

StructField("col2", DoubleType()),

StructField("cat1", StringType()),

# ...

])

df_test = spark.createDataFrame(pdf_test, schema=schema)

Accept as a solution if this helps!

Re: Cannot convert numpy type object to spark type

deborshi_nag — Tue, 23 Dec 2025 12:29:28 GMT

Hi,

It is quite possible that Model expects multiple raw columns (pyfunc/predict on a table of primitives)

Then all columns in inputCols must be Spark primitive types (e.g., DoubleType, IntegerType, StringType, BooleanType). Avoid object, mixed arrays, or Python objects.

from pyspark.sql import functions as F

from pyspark.sql.types import DoubleType, IntegerType, StringType, BooleanType, FloatType

# Cast numerics to DoubleType (safer for Arrow/pandas UDFs)

for c, t in df_test.dtypes:

if t in ('tinyint', 'smallint', 'int', 'bigint', 'float', 'double', 'decimal'):

df_test = df_test.withColumn(c, F.col(c).cast(DoubleType())

It can also mean that somewhere in the pipeline Spark is seeing a NumPy object dtype (from pandas or an Arrow conversion) and doesn’t know how to map it to a Spark type.

# If you accidentally used pandas:

import pandas as pd

from pyspark.sql.types import *

# Suppose you have a pandas DataFrame `pdf_test`

# Clean up pandas object dtypes to concrete types first

for col in pdf_test.columns:

if pdf_test[col].dtype == 'object':

# If it's actually numeric but stored as object

try:

pdf_test[col] = pd.to_numeric(pdf_test[col])

except Exception:

# Otherwise keep as string

pdf_test[col] = pdf_test[col].astype(str)

# Then define a schema (recommended)

schema = StructType([

# Adjust types to match your data and your model signature

StructField("col1", DoubleType()),

StructField("col2", DoubleType()),

StructField("cat1", StringType()),

# ...

])

df_test = spark.createDataFrame(pdf_test, schema=schema)

Please accept it as a solution if it helped!