Forum Discussion

dbeavon3's avatar
dbeavon3
Icon for Memorable Member rankMemorable Member
10 months ago

PySpark Notebook is Killed after Encountering Bad Gateway (localhost:8998)

Here is yet another crazy error in Fabric Spark...   LivyHttpRequestFailure: Submission failed due to error content =[{"error":{"code":"BadGateway","message":"Upstream service is unreachable","inn...
data engineering
tayloramy's avatar
tayloramy
Icon for Super User rankSuper User
10 months ago

Hi dbeavon3, 

 

I understand your frustration here, something isn't working and at the very least the error message is useless. 

 

I suggest opening a ticket with Microsoft as they will be able to dig into the logs more and help, we can only gives suggestions based on our experience here. 

 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution. 

v-menakakota's avatar
v-menakakota
Icon for Community Support rankCommunity Support
10 months ago

Hi  dbeavon3  ,
Thanks for reaching out to the Microsoft fabric community forum.

 

I would also take a moment to thank  tayloramy  , for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.

Could you please consider raising a support ticket with Microsoft Fabric so the product team can review the internal logs and identify the root cause of the LivyHttpRequestFailure and BadGateway (localhost:8998) error?

Please refer below link on how to raise a contact support or support ticket. 
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn 

 

Regards,
Microsoft Fabric Community Support Team.

  • v-menakakota's avatar
    v-menakakota
    Icon for Community Support rankCommunity Support
    10 months ago

    Hi dbeavon3 ,

    We are following up once again regarding your query. Could you please confirm if the issue has been resolved through the support ticket with Microsoft? 

    If the issue has been resolved, we kindly request you to share the resolution or key insights here to help others in the community. 

     

    Thank you

    • dbeavon3's avatar
      dbeavon3
      Icon for Memorable Member rankMemorable Member
      10 months ago

      Hi v-menakakota 

      This posting was created as a "placeholder", in order to gather more information in this community.  Nobody has yet explained why Livy is used mid-way thru a long-running notebook.  Nobody explained what this exception is for (LivyHttpRequestFailure).  Nobody explained what the "gateway" does (localhost:8998).  Nobody shared the responsibilities of the "pubsub" agent (Microsoft.Analytics.Clusters.Services.PubSubAgent).

       

      Tayloramy explained that it is "all internal stuff".

       

      But obviously that isn't useful to me or to the next person who encounters this sort of failure.

       

      We should think of this post as a "living wiki", and maybe some day it will develop into a full-blown MT TSG ("trouble shooting guide").  Over time others may share more details about what is happening with the livy request failures.  Those are actually quite COMMON, and you can find references to them if you care to search.  Unfortunately those google results don't have the closure we need.  But users of fabric seem very patient, and very happy to experience the types of failures that only happen 1% or 2% of the time.  Using retries seems to be the magical fix for every problem and we sprinkle it everywhere -  like pixie dust - in order to try to avoid these strange bugs we find in fabric.