Forum Discussion
Memorable MemberPySpark Notebook is Killed after Encountering Bad Gateway (localhost:8998)
Super UserHi dbeavon3
Here are two ways I see an error in your Pyspark Notebook.
1. You are using Unmanaged Tables in Tables ( data + separated metadata ), metadata is still in the system in Delta Lake.
Use Files instead of Tables.
2. Use managed Tables instead ( data + metadata altogether )
SaveAsTable() command in Delta Lake.
So you can delete the entire files system and Use managed tables. That way we can see cost, time and energy.
Memorable MemberHow is it that you consider the external assets to be a so-called "error"? You should support with an authoritative link. Obviously I have many blogs and articles that say unmanaged tables are fully supported.
In any case, could we please focus on the LivyHttpRequestFailure and the bad gateway? What does that mean?
- BhaveshPatel
Hi dbeavon3
I have tried this in my own workspace and this is the reason of my recommendation. There is no issue with Livy Server & bad gateway etc.. GET, POST those REST APIs
So Use managed Tables instead ( data + metadata altogether )
SaveAsTable() command in Delta Lake/ Fabric and I can clearly see the issue.
Check my blog : https://medium.com/@bhaveshpatelaus- dbeavon3
I'm looking for a technical explanation about a specific error message.
Fabric gives lots of random error messages, and if we spent two days re-architecting our soluctions every time we ran into an error then we would be constantly re-writing everything from scratch every week. I'd rather focus on the errors themselves and make sure we understand their meaning if possible.
- BhaveshPatel
Hi dbeavon3
How it is possible that you are having an error messages while For others are working fine with Delta Lake/Direct Lake etc..it just an REST APIs