topic Re: Spark Job Definition: Spark_Ambiguous_NonJvmUserApp_FailedContainerLaunch in Data Engineering

Spark Job Definition: Spark_Ambiguous_NonJvmUserApp_FailedContainerLaunch

DennesTorres — Mon, 02 Oct 2023 20:35:22 GMT

Hi,

I created a notebook which runs sucessfully on my lakehouse.

After that, I created a spark job definition, downloaded the notebooks in format .py, uploaded to the job definition and tested. Of course, I linked the lakehouse to the job definition, the same lakehouse.

I'm getting the error Spark_Ambiguous_NonJvmUserApp_FailedContainerLaunch

I couldn't find reference to this error anywhere. What does it means? It seems like I missed some step related to authentication, but I couldn't identify what.

Thank you in advance!

Kind Regards,

Dennes

Re: Spark Job Definition: Spark_Ambiguous_NonJvmUserApp_FailedContainerLaunch

DennesTorres — Tue, 03 Oct 2023 12:41:11 GMT

Hi,

The innitial message made me think the execution was not started, but going deep on the logs, I discovered the execution started and failed. The trick is that when using a notebook, "spark" is acessible as the current spark session, but when running a spark job, it's not.

As a result, the spark job code needs more details. In this case, a statement to create the session in a variable called "spark":

spark = SparkSession.builder \

.master("local[1]") \

.appName("SparkByExamples.com") \

.getOrCreate()

Kind Regards,

Dennes

Re: Spark Job Definition: Spark_Ambiguous_NonJvmUserApp_FailedContainerLaunch

Anonymous — Tue, 03 Oct 2023 13:07:32 GMT

Hi DennesTorres - Thanks for using Fabric Community,

As I understand you are trying to run/schedule notebook code using Spark Job Definition in MS Fabric and facing an error.

You might be facing this error Spark_Ambiguous_NonJvmUserApp_FailedContainerLaunch because your Spark job definition is missing a Spark session.

When you run the code in a notebook, Apache Spark automatically creates a Spark session. However, when you run the same code in a Spark job definition, you need to explicitly create a Spark session.
In addition, you need to avoid using the display() function in your Spark job definition, as it is also not supported.

To fix the error, you need to add the following code to the beginning of your Spark job definition:

from pyspark.sql import SparkSession spark = SparkSession.builder.getOrCreate()

Once you have added this code, your Spark job definition should run successfully.

You can also find your issues for Spark Job Definition here:

Hope this helps, Please do let us know if you have further queries.