Forum Discussion
Microsoft EmployeeR Model training takes forever in Fabric Notebook
Hi roshneematlani,
Currently there is no single official wiki page that explicitly calls out the limitation. But from the official Spark MLlib and SparkR documentation, we can see the set of supported GLM algorithms and parameters, and the bias reduced methods like Jeffreys/Firth are not part of that supported surface area. Please refer below documents:
https://spark.apache.org/docs/latest/ml-classification-regressionhttps://spark.apache.org/docs/latest/api/R/articles/sparkr-vignettes
Thanks and regards,
Anjan Kumar Chippa
Community SupportHi roshneematlani,
Thank you for reaching out to Microsoft Fabric Community.
This is not a known issue with GLM support in Fabric. While SparkR is supported in fabric, GLM training via glm() in SparkR is driver bound and is not distributed. Increasing fabric capacity only adds executor resources, which does not improve SparkR GLM performance.
In fabric notebooks long running SparkR jobs can be very slow and may hit notebook session or driver limits, that causes the disconnections you are seeing. This behaviour differs from databricks, where SparkR clusters allow larger and more persistent drivers.
For large datasets or multi-hour training, the recommend way is to use Spark MLlib GLM (via sparklyr or PySpark), which is fully distributed and supported in fabric. SparkR is best for the smaller or exploratory workloads.
Thanks and regards,
Anjan Kumar Chippa