Forum Discussion

sreedharshan_10's avatar
sreedharshan_10
Frequent Visitor
1 year ago
Solved

Livy HTTP Request Error

I am running a large-scale Spark job in Microsoft Fabric via Livy, where data is read from Delta tables and processed in iterative steps, with periodic checkpoints every few iterations to persist int...
Spark Job Definition
  • v-aatheeque's avatar
    v-aatheeque
    1 year ago

    Hi sreedharshan_10 

    Thanks for sharing the update. Since the job succeeds when run under a different account, the issue seems tied to resource load or background operations in the original dev account rather than Livy itself. Switching accounts works as a workaround, but it’s not the only option.

     

    A better long-term approach would be to:

    • Reviewing Fabric capacity metrics to check if background jobs are consuming resources.

    • Coordinating with your admin to isolate heavy Spark workloads onto a dedicated capacity or workspace.

    • Adjusting background activity in the dev account to reduce contention.

    Hope it helps !!

    Thank You.

     

v-aatheeque's avatar
v-aatheeque
Icon for Community Support rankCommunity Support
1 year ago

Hi sreedharshan_10 

I wanted to follow up regarding the LivyHttpRequestFailure error you mentioned earlier, specifically the RequestCancelled: upstreamService:livy, timeout:30s issue.

 

In the previous response, I shared a set of recommendations, such as increasing the timeout beyond 30 seconds, adjusting checkpoint intervals to reduce overhead, checking for networking or firewall issues, ensuring sufficient resources for Spark, using the Spark UI to identify long-running stages, upgrading to the latest stable Livy/Spark versions, and testing on a smaller dataset to isolate volume-related issues.

 

Were you able to try any of these steps in your environment? If so, did they help reduce the timeout errors or improve job stability? If the problem persists, let us know we can assist you.

v-aatheeque's avatar
v-aatheeque
Icon for Community Support rankCommunity Support
1 year ago

Hi sreedharshan_10 

 

I’m following up on the LivyHttpRequestFailure (RequestCancelled: upstreamService:livy, timeout:30s) issue you raised. In my earlier response, I suggested steps such as extending the timeout, tuning checkpoint intervals, verifying networking/firewall settings, checking Spark resource availability, reviewing long-running stages in the Spark UI, upgrading to the latest stable versions, and testing with smaller datasets.

Have you had a chance to try these recommendations? If so, did they help reduce the timeouts or improve stability?

If the issue remains, please let us know so we can continue troubleshooting with you.