Forum Discussion
Livy Error with runMultiple() DAG Chunking - InvalidHttpRequestToLivy: from cannot be less than 0 HT
The solution provided by the product team was to place this line before the runMultiple to prevent the DAG orchestration from displaying its graphics. It seems to be working and they said a full fix would be implemented soon.
spark.conf.set("spark.notebookutils.run.progressbar.enabled", "false")
Instead of reusing the same session for every runMultiple batch, try Create a new Livy session per DAG chunk. This avoids log cursor conflicts and buffer overflow across chunks.
When you say “create a new Livy session per DAG chunk,” do you mean there’s a PySpark or Fabric-native command I can run inside the same notebook to reset/reinitialize the Livy session—before calling runMultiple() again?
Just to clarify, my current setup is entirely notebook-driven:
---One orchestration notebook.
---It uses nbutils.runMultiple() inside a for loop to submit batches (chunks).
---Each runMultiple() call executes a batch of worker notebooks in parallel.
There’s no pipeline orchestration around it, so the notebook session itself stays open while runMultiple() runs multiple times.
Did you have any specific detail for this? I believe the livy session and the notebook session can't be seperated. When I forced a restart of the livy session the notebook stopped.