Forum Discussion
Memorable MemberAnyone having luck with pyspark workloads in Fabric? Getting assorted error messages.
Hi dbeavon3,
From Error-2, I see you are using native engine on spark. Did you try turning it off and running the same notebook? (https://learn.microsoft.com/en-us/fabric/data-engineering/native-execution-engine-overview?tabs=sparksql#enable-for-a-notebook-or-spark-job-definition)
From all the error messages, I can offer you a speculative explanation. When Spark runs, VMs get spun up with specific configuration which run as Executors. I believe for some reason a VM had crashed (most likely because of running a unsupported query on native engine). And once the VM crashed, Spark application also crashed. Usually VM failures are automatically managed by Spark, but I guess with native engine integration, it still needs improvement from Microsoft.
Community SupportHi,
Thank you for reaching out to the MS Fabric community forum.
I understand that you are encountering unfamiliar errors. Let's go through each of the errors you've mentioned:
Error 1: invalid_grant: Error(s): 501481 This occurs when opening logs from the Spark UI. It means that the code verifier and code challenge in the authorization request do not match. Ensure that these values are correctly configured and try regenerating them.
Error 2: LD_PRELOAD warnings Warnings: that a shared object file cannot be preloaded (/opt/gluten/dep/libjemalloc.so.2). Known issue, just ignore it. To avoid these warnings, set LD_PRELOAD settings in your Spark configuration.
Error 3: java.lang.reflect.UndeclaredThrowableException This error is encountered when trying to create a connection in the Livy notebook because of a ClassNotFoundException for org.apache.spark.shuffle.sort.ColumnarShuffleManager. Make sure all the necessary dependencies are in your Spark environment and add the missing library or jar file.
Error 4: ExecutorMonitor threw an exception. The null pointer exception in the Executor monitor is an issue with the dynamic allocation of executors. Remove dynamic allocation or upgrade your Spark version.
I hope this helps resolve the issues you're experiencing. Should the problems continue, please consider raising a Microsoft support ticket for further assistance. Here is the link: https://learn.microsoft.com/en-us/power-bi/support/create-support-ticket
If this helps then please Accept it as a solution and dropping a "Kudos" so other members can find it more easily.
Thanks.