Livy session becomes dead when reading large file in Fabric Notebook

[gbetancurm]

Hi everyone,

I'm running a notebook in Microsoft Fabric where I read a large file, around 2GB.

The notebook takes a long time to execute, and eventually I receive the following error:

InvalidHttpRequestToLivy: Session -------- of workspace --------- on pool -------- is not running. It is either completed or not fully yet ready. Scheduler state : Scheduled. Plugin state : Cleanup. Livy state : dead HTTP status code: 400. Trace ID: --------.

My question is: Can this issue be handled from the code side, or is it more likely related to Spark/Livy configuration settings? What would be the best way to fix it?

Has anyone experienced something similar when processing large files in Fabric?

Any guidance would be greatly appreciated.

Thanks in advance!

[v-veshwara-msft]

Hi @gbetancurm ,

We wanted to kindly follow up regarding your query. If you need any further assistance, please reach out.
Thank you.

[v-veshwara-msft]

Hi @gbetancurm ,

Thanks for reaching out to Microsoft Fabric Community.

Just checking in to see if the response provided by @deborshi_nag  was helpful.

If you have reviewed the Spark application Driver logs as suggested and still need further assistance, please feel free to share additional details so we can assist further.

Thank you.

[joaotinti]

Same error here, but in synapse analytics.

[deborshi_nag · Accepted solution]

Hello @gbetancurm 

It is probably a driver out-of-memory issue, which would happen if you're using Pandas dataframe instead of Spark. Pandas is not distributed by design, so none of the executors are used and the driver gets overloaded. 

You can start by looking at the Spark application details > Logs > Driver - you should find additional details on why the application failed.