Forum Discussion
Cannot convert numpy type object to spark type
I am new to MS fabric and trying to do a tutorial hands on - https://learn.microsoft.com/en-us/fabric/data-science/tutorial-data-science-batch-scoring I am running the following code to Instantia...
Hello
This error usually comes from a type mismatch between what MLFlowTransformer (and the underlying pandas UDF it uses) expects and what your DataFrame is actually providing.
ValueError: Cannot convert numpy type object to spark type typically means that somewhere in the pipeline Spark is seeing a NumPy object dtype (from pandas or an Arrow conversion) and doesn’t know how to map it to a Spark type. This often happens when:
- You pass a pandas DataFrame (or columns derived from pandas with object dtype) instead of a Spark DataFrame.
- Your Spark columns contain arrays, maps, or mixed-type columns that Spark inferred as object (via Arrow/pandas), or they’re null-heavy and Spark guessed an ambiguous type.
- The MLflow model signature expects certain columns/types (e.g., a single features vector) but you’re passing raw columns as inputCols that include non-primitive types or types that don’t match the model signature.
# If you accidentally used pandas:
import pandas as pd
from pyspark.sql.types import *
# Suppose you have a pandas DataFrame `pdf_test`
# Clean up pandas object dtypes to concrete types first
for col in pdf_test.columns:
if pdf_test[col].dtype == 'object':
# If it's actually numeric but stored as object
try:
pdf_test[col] = pd.to_numeric(pdf_test[col])
except Exception:
# Otherwise keep as string
pdf_test[col] = pdf_test[col].astype(str)
# Then define a schema (recommended)
schema = StructType([
# Adjust types to match your data and your model signature
StructField("col1", DoubleType()),
StructField("col2", DoubleType()),
StructField("cat1", StringType()),
# ...
])
df_test = spark.createDataFrame(pdf_test, schema=schema)
Accept as a solution if this helps!