Pyspark code running twice causing a LIVY status = DEAD

Hi mkj1213 

I feel like the number of Jobs doesn't mean that the code has run the same number of times. I made some test based on your code (have made some modifications to fit my data). In the following images, you will see that it experienced 7 Spark jobs. 

When I query data from the result delta table, it doesn't show any duplicated rows. This means the code runs only once although it shows 7 Spark jobs. 

According to my research, this "LIVY status = DEAD" error is more like a result of a lack of some kind of resource or a resource that goes out of limit. I find an Azure Synapse Analytics Blog related to this error as below. According to the solution in it, you can try to increase the node size of the Spark pool which is used to run the notebook. 

Livy is dead and some logs to help. - Microsoft Community Hub

Here are some Fabric documentations about setting Spark pool:

Workspace administration settings in Microsoft Fabric - Microsoft Fabric | Microsoft Learn

Manage settings for data engineering and science capacity - Microsoft Fabric | Microsoft Learn

Hope this will be helpful!

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!