Forum Discussion
Memorable MemberAnyone having luck with pyspark workloads in Fabric? Getting assorted error messages.
Hi dbeavon3,
From Error-2, I see you are using native engine on spark. Did you try turning it off and running the same notebook? (https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview?tabs=sparksql#enable-for-a-notebook-or-spark-job-definition)
From all the error messages, I can offer you a speculative explanation. When Spark runs, VMs get spun up with specific configuration which run as Executors. I believe for some reason a VM had crashed (most likely because of running a unsupported query on native engine). And once the VM crashed, Spark application also crashed. Usually VM failures are automatically managed by Spark, but I guess with native engine integration, it still needs improvement from Microsoft.
Memorable MemberHi govindarajan_d
I'm new to the Fabric implementation of Spark and was worried about all these proprietary components and error messages.
I've only been using if one week. I only had one day where these unfamiliar errors were appearing. However it was on my very first day with Spark in Fabric ... so that is what made me concerned. Since then, we have not seen it repeated. However, I wanted to ask for tips in preparation for the next time we see these errors. Else I will be no better off than I was on day one.
I have a speculative explanation as well. The weird/unusual thing about Spark in Fabric is the integration with Entra ID user credentials. In the other Spark environments which I've used, the cluster was always running as a system-level account (OS account or service principal). However I think that the Spark notebooks in Fabric are constantly referring to Entra ID in order to validate the PBI user's prior credentials, or retrieve new credentials. This introduces code that may be (1) a weak link, and (2) very proprietary and very different than what is found in the OSS spark implementation.
... this theory would also explain the presence of these strange error messages which aren't able to be found in google. I may be the first person ever to post their error messages on the Internet!
Super UserHi dbeavon3,
You are right. Microsoft has implemented proprietary code on top of OSS Spark and that makes it a source for error messages that are uncommon for people who worked with OSS Spark. Microsoft has to add more informative error messages!