Forum Discussion

dominikh's avatar
dominikh
Frequent Visitor
2 years ago

Error configuring spark session with magic command in pipeline activity

Hi there,

 

I'm trying to configure my spark session in a Fabric notebook using the magic command detailed here: https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#spark-session-configuration-magic-command

 

I only need to configure the default lakehouse, which runs successfully in the notebook interactive mode:

 

The issue is that when I run this notebook in a pipeline activity, I get an error:

 

Details:

Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Error while creating the livy session, RunId: 1db415da-c5cf-482b-b787-40311575ef19' :

 

Any ideas on what I'm doing wrong here? Help would be much appreciated,

Dominik

data engineering

6 Replies

  • dominikh's avatar
    dominikh
    Frequent Visitor
    2 years ago

    Hi govindarajan_d,

     

    Thank you for trying to reproduce the issue as well. I got this issue yesterday for the first time, so it has persisted for a day at least. I'll give it another try tomorrow and see what happens. I'll also check for any running sessions and end them if needed. 

  • HimanshuS-msft's avatar
    HimanshuS-msft
    Microsoft Employee
    2 years ago

    Hello dominikh , 
    I also suggest you to add the retry and retry interval option set and see if that helps . I have seen this error but for me its not consistent .


    thanks 
    Himanshu 

    • dominikh's avatar
      dominikh
      Frequent Visitor
      2 years ago

      Hi HimanshuS-msft,

       

      Thank you for the suggestion. It hasn't helped unfortunately. 😕 I configured the notebook activity with 3 retries and they all failed with the same error:

       

      Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Error while creating the livy session

       

       

      Where can I find the run logs on the notebook? When I go to 'More options' > 'Recent runs' I don't see this pipeline run above. 

       

      Sincerely,

      Dominik