Forum Discussion
Microsoft EmployeeR Model training takes forever in Fabric Notebook
Hi roshneematlani,
Currently there is no single official wiki page that explicitly calls out the limitation. But from the official Spark MLlib and SparkR documentation, we can see the set of supported GLM algorithms and parameters, and the bias reduced methods like Jeffreys/Firth are not part of that supported surface area. Please refer below documents:
https://spark.apache.org/docs/latest/ml-classification-regressionhttps://spark.apache.org/docs/latest/api/R/articles/sparkr-vignettes
Thanks and regards,
Anjan Kumar Chippa
Microsoft EmployeeMy current model is as follows-
fit <- glm( formula = y~ x1+ x2+ x3+ x4 + x5 + x6 + x7 + x8 + x9 + x10, data = train_df, family = binomial, weights = weights, method = "brglmFit", type = 'MPL_Jeffreys', a = 0.1, control = list( maxit = 5000, trace = TRUE, slowit = 0.1 ) )
and based on my research- using
method = "brglmFit"
type = "MPL_Jeffreys"
in fabric sparkR is very slow and equivalent is NOT available in Spark MLlib.
Any other alternative for fabric?
Train brglmFit model locally (CPU/GPU)
Save model coefficients
Load them into:
Django backend
Or inference-only service
Fabric is then used for:
Data ingestion
Feature prep
Analytics