Forum Discussion

roshneematlani's avatar
roshneematlani
Icon for Microsoft Employee rankMicrosoft Employee
7 months ago
Solved

R Model training takes forever in Fabric Notebook

Hello,   I use GLM Model in R in databricks which usually takes 3-4 hrs for model training. I am trying to replicate same in the Fabric notebook in SparkR, and it is taking forver to run. Everytime...
data science
ml model
notebook
v-achippa's avatar
v-achippa
Icon for Community Support rankCommunity Support
7 months ago

Hi roshneematlani,

 

Yes you are correct, brglmFit with MPL_Jeffreys has no equivalent in Spark MLlib, and that is why the workload performs poorly in fabric sparkR. These models are strictly single node and run on the spark driver, because of that long running training disconnects in fabric.

There is no equivalent or better alternative inside fabric today.

If jeffreys prior bias reduction is required, the recommended approach is to train the model outside fabric using native R and then deploy the trained coefficients for scoring inside fabric.

Thanks and regards,

Anjan Kumar Chippa

v-achippa's avatar
v-achippa
Icon for Community Support rankCommunity Support
7 months ago

Hi roshneematlani,

 

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution I have provided for the issue worked? or let us know if you need any further assistance.

 

Thanks and regards,

Anjan Kumar Chippa

  • roshneematlani's avatar
    roshneematlani
    Icon for Microsoft Employee rankMicrosoft Employee
    7 months ago

    Do we have any official wiki or any other resource which states this limitation?

     

    Thanks

    Roshnee