Forum Discussion

gbetancurm's avatar
gbetancurm
New Member
6 months ago
Solved

Livy session becomes dead when reading large file in Fabric Notebook

Hi everyone,

I'm running a notebook in Microsoft Fabric where I read a large file, around 2GB.

The notebook takes a long time to execute, and eventually I receive the following error:

InvalidHttpRequestToLivy: Session -------- of workspace --------- on pool -------- is not running. It is either completed or not fully yet ready. Scheduler state : Scheduled. Plugin state : Cleanup. Livy state : dead HTTP status code: 400. Trace ID: --------.

My question is: Can this issue be handled from the code side, or is it more likely related to Spark/Livy configuration settings? What would be the best way to fix it?

Has anyone experienced something similar when processing large files in Fabric?

Any guidance would be greatly appreciated.

Thanks in advance!

data engineering
environment
notebook
  • deborshi_nag's avatar
    deborshi_nag
    6 months ago

    Hello gbetancurm 

     

    It is probably a driver out-of-memory issue, which would happen if you're using Pandas dataframe instead of Spark. Pandas is not distributed by design, so none of the executors are used and the driver gets overloaded. 

     

    You can start by looking at the Spark application details > Logs > Driver - you should find additional details on why the application failed. 

     

4 Replies

  • deborshi_nag's avatar
    deborshi_nag
    Super User
    6 months ago

    Hello gbetancurm 

     

    It is probably a driver out-of-memory issue, which would happen if you're using Pandas dataframe instead of Spark. Pandas is not distributed by design, so none of the executors are used and the driver gets overloaded. 

     

    You can start by looking at the Spark application details > Logs > Driver - you should find additional details on why the application failed. 

     

  • v-veshwara-msft's avatar
    v-veshwara-msft
    Community Support
    6 months ago

    Hi gbetancurm ,

    Thanks for reaching out to Microsoft Fabric Community.

     

    Just checking in to see if the response provided by deborshi_nag  was helpful.

    If you have reviewed the Spark application Driver logs as suggested and still need further assistance, please feel free to share additional details so we can assist further.

     

    Thank you.

  • v-veshwara-msft's avatar
    v-veshwara-msft
    Community Support
    6 months ago

    Hi gbetancurm ,

    We wanted to kindly follow up regarding your query. If you need any further assistance, please reach out.
    Thank you.