Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified in Microsoft Fabric—for free! For a limited time, the Microsoft Fabric Community team will be offering free DP-600 exam vouchers. Prepare now

Reply
dominikh
Frequent Visitor

Error configuring spark session with magic command in pipeline activity

Hi there,

 

I'm trying to configure my spark session in a Fabric notebook using the magic command detailed here: https://learn.microsoft.com/en-us/fabric/data-engineering/author-execute-notebook#spark-session-conf...

 

I only need to configure the default lakehouse, which runs successfully in the notebook interactive mode:

dominikh_0-1703260684333.png

 

The issue is that when I run this notebook in a pipeline activity, I get an error:

dominikh_1-1703260782378.png

 

Details:

Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Error while creating the livy session, RunId: 1db415da-c5cf-482b-b787-40311575ef19' :

 

Any ideas on what I'm doing wrong here? Help would be much appreciated,

Dominik

6 REPLIES 6
HimanshuS-msft
Community Support
Community Support

Hello @dominikh , 
I also suggest you to add the retry and retry interval option set and see if that helps . I have seen this error but for me its not consistent .

HimanshuSmsft_0-1703281153149.png


thanks 
Himanshu 

Hi @HimanshuS-msft,

 

Thank you for the suggestion. It hasn't helped unfortunately. 😕 I configured the notebook activity with 3 retries and they all failed with the same error:

 

Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Exception, Error value - Error while creating the livy session

 

dominikh_0-1703685738774.png

 

Where can I find the run logs on the notebook? When I go to 'More options' > 'Recent runs' I don't see this pipeline run above. 

 

Sincerely,

Dominik

Hi @dominikh 
Apologies for the issue you have been facing. Are you still facing this issue?
If yes I would ask you to raise a support ticket here. https://support.fabric.microsoft.com/en-US/support/
Please provide the details here for us to track internally.
Thanks

Hello @v-nikhilan-msft, thanks for the recommendation. I am still facing this issue, so I will raise a support ticket. 

dominikh
Frequent Visitor

Hi @govindarajan_d,

 

Thank you for trying to reproduce the issue as well. I got this issue yesterday for the first time, so it has persisted for a day at least. I'll give it another try tomorrow and see what happens. I'll also check for any running sessions and end them if needed. 

govindarajan_d
Solution Supplier
Solution Supplier

@dominikh I tried to reproduce the issue and got the same error.

Seems to have happened in old Synapse as well. Might be a temp issue as mentioned in the below link:

https://learn.microsoft.com/en-us/answers/questions/1125423/synapse-error-failed-to-create-livy-sess...

Helpful resources

Announcements
Oct Fabric Update Carousel

Fabric Monthly Update - October 2024

Check out the October 2024 Fabric update to learn about new features.

September Hackathon Carousel

Microsoft Fabric & AI Learning Hackathon

Learn from experts, get hands-on experience, and win awesome prizes.

October NL Carousel

Fabric Community Update - October 2024

Find out what's new and trending in the Fabric Community.