topic Re: Unable to load ML model in Data Science

Unable to load ML model

abm_chris — Thu, 06 Jun 2024 14:29:58 GMT

I've tried over the past couple of days to get ML models working in Fabric Notebooks with MLFlow using the MS Learn document here. I have tried both using the transformer API as well as loading through a Lakehouse table in the UI and keep getting the same error:

"RuntimeError: Unable to load model: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext"

The model definitely exists within the same workspace I'm accessing and the call appears to work when using

model = MLFlowTransformer() function.

Is there something else that I may be missing where it is erroring due to something else as I'd really like to be able to proceed with this as I've got tons of ideas I can use the platform for this.

Re: Unable to load ML model

Anonymous — Fri, 07 Jun 2024 12:40:14 GMT

Hi abm_chris ,

Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

Re: Unable to load ML model

abm_chris — Fri, 07 Jun 2024 13:50:27 GMT

The main issue I had was using this particular function:
from synapse.ml.predict import MLFlowTransformer

I've since used a different method available at the below video around 21-23 minutes which has been less troublesome and I've been able to put together what I was after:

Fabric Data Science Sales Forecasting: Predict sales numbers for product categories at a superstore. (youtube.com)

Re: Unable to load ML model

Anonymous — Mon, 10 Jun 2024 11:28:28 GMT

Hi abm_chris ,

I am unable to understand your previous reply and was just checking back to see if you have a resolution yet .

Thank you

Re: Unable to load ML model

Anonymous — Tue, 11 Jun 2024 10:39:56 GMT

Hi abm_chris ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
Otherwise, will respond back with the more details and we will try to help .
Thanks

Re: Unable to load ML model

LukeHe — Wed, 10 Jul 2024 11:38:26 GMT

I think what they're saying is that they've found another totally separate resource (That video link) that fixes the missing information in the documentation that's causing the error:

ImportError Traceback (most recent call last) Cell In[13], line 3 1 #Test the model on new sample data points: 2 # Inference with loading the logged model ----> 3 from synapse.ml.predict import MLflowTransformer 5 spark.conf.set("spark.synapse.ml.predict.enabled", "true") 7 model = MLflowTransformer( 8 inputCols=["x"], 9 outputCol="prediction", 10 modelName="sample-sklearn", 11 modelVersion=1, 12 ) ImportError: cannot import name 'MLflowTransformer' from 'synapse.ml.predict' (/home/trusted-service-user/cluster-env/trident_env/lib/python3.10/site-packages/synapse/ml/predict/__init__.py)

I'm experiencing the same error when simply following the tutorial steps exactly. I'm surprised that you've not been able to answer the initial question in a more satisfactory way (Ie, actually provide a reason and suggested resolution) especially considering this is marked as resolved when from my perspective it totally isn't.

Re: Unable to load ML model

LukeHe — Wed, 10 Jul 2024 11:39:57 GMT

To restate the question, the code sample provided results in the error saying that it cannot import name 'MLflowTransformer' from 'synapse.ml.predict'. The question is why is this and what is the proposed fix?