Forum Discussion
Cannot convert numpy type object to spark type
I am new to MS fabric and trying to do a tutorial hands on - https://learn.microsoft.com/en-us/fabric/data-science/tutorial-data-science-batch-scoring I am running the following code to Instantia...
Hi,
It is quite possible that Model expects multiple raw columns (pyfunc/predict on a table of primitives)
Then all columns in inputCols must be Spark primitive types (e.g., DoubleType, IntegerType, StringType, BooleanType). Avoid object, mixed arrays, or Python objects.
from pyspark.sql import functions as F
from pyspark.sql.types import DoubleType, IntegerType, StringType, BooleanType, FloatType
# Cast numerics to DoubleType (safer for Arrow/pandas UDFs)
for c, t in df_test.dtypes:
if t in ('tinyint', 'smallint', 'int', 'bigint', 'float', 'double', 'decimal'):
df_test = df_test.withColumn(c, F.col(c).cast(DoubleType())
It can also mean that somewhere in the pipeline Spark is seeing a NumPy object dtype (from pandas or an Arrow conversion) and doesn’t know how to map it to a Spark type.
# If you accidentally used pandas:
import pandas as pd
from pyspark.sql.types import *
# Suppose you have a pandas DataFrame `pdf_test`
# Clean up pandas object dtypes to concrete types first
for col in pdf_test.columns:
if pdf_test[col].dtype == 'object':
# If it's actually numeric but stored as object
try:
pdf_test[col] = pd.to_numeric(pdf_test[col])
except Exception:
# Otherwise keep as string
pdf_test[col] = pdf_test[col].astype(str)
# Then define a schema (recommended)
schema = StructType([
# Adjust types to match your data and your model signature
StructField("col1", DoubleType()),
StructField("col2", DoubleType()),
StructField("cat1", StringType()),
# ...
])
df_test = spark.createDataFrame(pdf_test, schema=schema)
Please accept it as a solution if it helped!