Error configuring spark session with magic command in pipeline activity

[dominikh]

Hi there,

I'm trying to configure my spark session in a Fabric notebook using the magic command detailed here: https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#spark-session-conf...

I only need to configure the default lakehouse, which runs successfully in the notebook interactive mode:

The issue is that when I run this notebook in a pipeline activity, I get an error:

Details:

Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Error while creating the livy session, RunId: 1db415da-c5cf-482b-b787-40311575ef19' :

Any ideas on what I'm doing wrong here? Help would be much appreciated,

Dominik

[HimanshuS-msft]

Hello @dominikh , 
I also suggest you to add the retry and retry interval option set and see if that helps . I have seen this error but for me its not consistent .

thanks 
Himanshu 

[dominikh]

Hi @HimanshuS-msft,

Thank you for the suggestion. It hasn't helped unfortunately. 😕 I configured the notebook activity with 3 retries and they all failed with the same error:

Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Error while creating the livy session

Where can I find the run logs on the notebook? When I go to 'More options' > 'Recent runs' I don't see this pipeline run above. 

Sincerely,

Dominik

[Anonymous]

Hi @dominikh 
Apologies for the issue you have been facing. Are you still facing this issue?
If yes I would ask you to raise a support ticket here. https://support.fabric.microsoft.com/en-US/support/
Please provide the details here for us to track internally.
Thanks

[dominikh]

Hello @Anonymous, thanks for the recommendation. I am still facing this issue, so I will raise a support ticket. 

[dominikh]

Hi @govindarajan_d,

Thank you for trying to reproduce the issue as well. I got this issue yesterday for the first time, so it has persisted for a day at least. I'll give it another try tomorrow and see what happens. I'll also check for any running sessions and end them if needed. 

[govindarajan_d]

@dominikh I tried to reproduce the issue and got the same error.

Seems to have happened in old Synapse as well. Might be a temp issue as mentioned in the below link:

https://learn.microsoft.com/en-us/answers/questions/1125423/synapse-error-failed-to-create-livy-sess...