topic Re: I am getting Livy Http Request Failure in Fabric Notebooks. in Data Engineering

I am getting Livy Http Request Failure in Fabric Notebooks.

Anushka96 — Mon, 05 May 2025 08:14:44 GMT

I am trying to migrate somefiles to ADLS Gen2 using a high concurrency notebook in pipeline using Fabric F64 Trial Capacity.Pipline got fail after 1d and 20 hours and I am getting following error.
LivyHttpRequestFailure: Something went wrong while processing your request. Please try again later. HTTP status code: 500.
What can be the reason for this?

Re: I am getting Livy Http Request Failure in Fabric Notebooks.

Srisakthi — Mon, 05 May 2025 08:54:10 GMT

Hi Anushka96 ,

By looking at the error message we cannot predict exactly the root cause of the issue, basically issue is due to spark is not able to handle the request(could be due to not enough CUs).

What is the size of your files?

How many files you are migrating and what kind of transformation logic you are using?

What is spark pool configurations you have setup?

Are you using latest runtime version?

Can you install MS capacity metric app and check the CUs consumption?

https://learn.microsoft.com/en-us/fabric/enterprise/metrics-app-install?tabs=1st

Regards,

Srisakthi

Re: I am getting Livy Http Request Failure in Fabric Notebooks.

Anushka96 — Mon, 05 May 2025 09:01:05 GMT

Hi Srisakthi , Not using any transformation logic it is just using sharepoint REST api and write in to ADLS Gen2 Blob.

Re: I am getting Livy Http Request Failure in Fabric Notebooks.

Srisakthi — Mon, 05 May 2025 09:04:22 GMT

Hi Anushka96 ,

Is there any specific requirement to use notebooks?. Have you tried data pipeline and land to one lake

Regards,

Srisakthi

Re: I am getting Livy Http Request Failure in Fabric Notebooks.

v-ssriganesh — Tue, 10 Jun 2025 11:51:02 GMT

Hi Anushka96,

Thank you for posting your query in the Microsoft Fabric Community Forum, and thanks to Srisakthi for sharing valuable insights.

I am following up to see if you had a chance to review previous response of srisakthi's and provide the requested information. This will enable us to assist you further.

Thank you.

Re: I am getting Livy Http Request Failure in Fabric Notebooks.

v-ssriganesh — Sun, 11 May 2025 16:31:33 GMT

Hi Anushka96,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.

Re: I am getting Livy Http Request Failure in Fabric Notebooks.

v-ssriganesh — Tue, 10 Jun 2025 11:38:35 GMT

Hi Anushka96,

May I ask if you have resolved this issue? If so, kindly share the insights, as this will assist other community members in resolving similar issues more efficiently.

Thankyou.

Re: I am getting Livy Http Request Failure in Fabric Notebooks.

v-ssriganesh — Wed, 11 Jun 2025 10:23:42 GMT

Hello Anushka96,

Based on your scenario using a high-concurrency notebook to migrate files, here are some possible causes:

F64 Trial capacity has resource limits and may timeout long-running jobs.
Making REST API calls inside Spark UDFs can lead to memory issues, driver bottlenecks, and Livy session failures.

Best practices that may help prevent this issue:

Avoid row-by-row API calls in UDFs they are inefficient and unstable at scale.
Use rdd.mapPartitions() to process data in batches and reuse sessions.
Create a single requests.Session() per partition to avoid connection overhead.
Add retry logic and error handling to prevent executor crashes.
Filter and batch data before making API calls, and split large jobs into smaller chunks.

If this information is helpful, please “Accept as solution” and give a "kudos" to assist other community members in resolving similar issues more efficiently.
Thank you.

Re: I am getting Livy Http Request Failure in Fabric Notebooks.

v-ssriganesh — Sat, 14 Jun 2025 17:41:41 GMT

Hi Anushka96,
I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please accept it as a solution and give it a 'Kudos' so other community members with similar problems can find a solution faster.
Thank you.