Forum Discussion
Pyspark code running twice causing a LIVY status = DEAD
Hi mkj1213
I feel like the number of Jobs doesn't mean that the code has run the same number of times. I made some test based on your code (have made some modifications to fit my data). In the following images, you will see that it experienced 7 Spark jobs.
When I query data from the result delta table, it doesn't show any duplicated rows. This means the code runs only once although it shows 7 Spark jobs.
According to my research, this "LIVY status = DEAD" error is more like a result of a lack of some kind of resource or a resource that goes out of limit. I find an Azure Synapse Analytics Blog related to this error as below. According to the solution in it, you can try to increase the node size of the Spark pool which is used to run the notebook.
Livy is dead and some logs to help. - Microsoft Community Hub
Here are some Fabric documentations about setting Spark pool:
Workspace administration settings in Microsoft Fabric - Microsoft Fabric | Microsoft Learn
Manage settings for data engineering and science capacity - Microsoft Fabric | Microsoft Learn
Hope this will be helpful!
Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!
Hi mkj1213
I feel like the number of Jobs doesn't mean that the code has run the same number of times. I made some test based on your code (have made some modifications to fit my data). In the following images, you will see that it experienced 7 Spark jobs.
When I query data from the result delta table, it doesn't show any duplicated rows. This means the code runs only once although it shows 7 Spark jobs.
According to my research, this "LIVY status = DEAD" error is more like a result of a lack of some kind of resource or a resource that goes out of limit. I find an Azure Synapse Analytics Blog related to this error as below. According to the solution in it, you can try to increase the node size of the Spark pool which is used to run the notebook.
Livy is dead and some logs to help. - Microsoft Community Hub
Here are some Fabric documentations about setting Spark pool:
Workspace administration settings in Microsoft Fabric - Microsoft Fabric | Microsoft Learn
Manage settings for data engineering and science capacity - Microsoft Fabric | Microsoft Learn
Hope this will be helpful!
Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!
Thanks for giving my problem some time on your end.
i will look into the links that you had provided in more details and get back to you. The only difference that i had noticed between the structure of your code and my code was the agg part. In my code i had made two aggregation (count and distinct count) while in mine you had only done one. Is it possible to add another aggregation method (maybe a count) and share the spark jobs screenshot (the second image that you had included in your comments.
On the spark pool settings, it turned out that i was using large size (maybe i could have increased it to X or XX size):
Regards
