Cannot convert numpy type object to spark type

The error you're seeing —
ValueError: Cannot convert numpy type object to spark type — typically means that Spark is encountering a data type in your input that it doesn't know how to convert into its internal types, and it often relates to using numpy.object_ or dtype=object in your DataFrame columns.

Root Cause:

In your code:

model = MLFlowTransformer(
    inputCols=list(df_test.columns),
    outputCol='predictions',
    modelName='lgbm_sm',
    modelVersion=1
)

You're passing list(df_test.columns) as the inputCols. This itself is likely not the problem.

The real issue is most likely with the df_test DataFrame — it seems to be a Pandas DataFrame (or a Spark DataFrame with some columns of ambiguous/numpy object type), and SynapseML expects a Spark DataFrame with well-defined schema and no ambiguous types.

✅ How to Fix It:

1. Ensure df_test is a Spark DataFrame, not a Pandas one. If it's currently a Pandas DataFrame, convert it:

spark_df_test = spark.createDataFrame(df_test)

But even when converting, Spark sometimes can't infer schema properly from object dtype. So it’s safer to:

1. Make sure df_test has clear types before conversion:

import pandas as pd

# Ensure all columns have concrete types (avoid object dtype)
df_test = df_test.astype({
    'col1': 'float64',
    'col2': 'int64',
    'col3': 'string',  # Replace with your actual column names/types
    # ...
})

Then convert:

spark_df_test = spark.createDataFrame(df_test)

1. Pass the Spark DataFrame to your pipeline:

model = MLFlowTransformer(
    inputCols=spark_df_test.columns,
    outputCol='predictions',
    modelName='lgbm_sm',
    modelVersion=1
)

transformed = model.transform(spark_df_test)

✅ Summary

• Ensure you're using a Spark DataFrame, not a Pandas one.

• Make sure all columns in your DataFrame have explicit types, especially no object types.

• Use .astype() in Pandas before converting to Spark DataFrame.

• Use spark.createDataFrame() for conversion.

Would you like help inferring the correct schema for your DataFrame or code to automatically clean the column types?