Forum Discussion

DennesTorres's avatar
DennesTorres
Power Participant
2 years ago
Solved

Spark Job Definition: Spark_Ambiguous_NonJvmUserApp_FailedContainerLaunch

Hi,   I created a notebook which runs sucessfully on my lakehouse.   After that, I created a spark job definition, downloaded the notebooks in format .py, uploaded to the job definition and teste...
data engineering
  • DennesTorres's avatar
    DennesTorres
    2 years ago

    Hi,

    The innitial message made me think the execution was not started, but going deep on the logs, I discovered the execution started and failed. The trick is that when using a notebook, "spark" is acessible as the current spark session, but when running a spark job, it's not.

    As a result, the spark job code needs more details. In this case, a statement to create the session in a variable called "spark":

    spark = SparkSession.builder   \
                .master("local[1]")   \
                        .appName("SparkByExamples.com") \
                        .getOrCreate()
     
    Kind Regards,
     
    Dennes
DennesTorres's avatar
DennesTorres
Power Participant
2 years ago

Hi,

The innitial message made me think the execution was not started, but going deep on the logs, I discovered the execution started and failed. The trick is that when using a notebook, "spark" is acessible as the current spark session, but when running a spark job, it's not.

As a result, the spark job code needs more details. In this case, a statement to create the session in a variable called "spark":

spark = SparkSession.builder   \
            .master("local[1]")   \
                    .appName("SparkByExamples.com") \
                    .getOrCreate()
 
Kind Regards,
 
Dennes