Forum Discussion
Cannot convert numpy type object to spark type
I ran into the same problem.
BTW, I used the notebook from Explore a sample in Fabric.
It turned out that the "Exited" column was converted into a string where the plot over the categorical columns was coded. Instead, the Exited column has to be of type integer before you feed it into the training.
You can check in your experiment in the MLmodel file:
In line 34 the dtype should be int64, not object!
Don't forget to adapt the version number to your latest version of the model in the call to MLFlowTransformer.
Hope it helps!
Michael
MiSchroe - Thanks for clarifying the cause of the error. Pardon my ignorance, but could you explain how you would change the dtype from object to int64 in the context of this data science tutorial. Any help would be greatly appreciated as I'd like to better understand how this all fits together and complete it.
In the notebook 3-train-evaluate I have changed the data type of the Exited column to int:
NOTE: In row 3 I load the table dbo.df_clean, because I have used a Lakehouse with the new schema. In the original tutorial the schema has to be omitted.
Great point. It solved my problem!
At first I went to convert all booleans to int, and of course it did not work. The problem was "Exited," as you mentioned already.
