Forum Discussion

dbeavon3's avatar
dbeavon3
Icon for Memorable Member rankMemorable Member
10 months ago

PySpark Notebook is Killed after Encountering Bad Gateway (localhost:8998)

Here is yet another crazy error in Fabric Spark...   LivyHttpRequestFailure: Submission failed due to error content =[{"error":{"code":"BadGateway","message":"Upstream service is unreachable","inn...
data engineering
BhaveshPatel's avatar
BhaveshPatel
Icon for Super User rankSuper User
10 months ago

Hi dbeavon3 

Here are two ways I see an error in your Pyspark Notebook. 

 

1. You are using Unmanaged Tables in Tables ( data + separated metadata ), metadata is still in the system in Delta Lake.

 

Use Files instead of Tables.  

 
Or 

2.  Use managed Tables instead ( data + metadata altogether )

SaveAsTable() command in Delta Lake.

 

So you can delete the entire files system and Use managed tables. That way we can see cost, time and energy.