Forum Discussion
Solved
Livy Error with runMultiple() DAG Chunking - InvalidHttpRequestToLivy: from cannot be less than 0 HT
I’m running into the Livy pagination bug that returns: InvalidHttpRequestToLivy: from cannot be less than 0 HTTP status code: 400. Setup: - Orchestrator notebook uses runMultiple() to start ...
The solution provided by the product team was to place this line before the runMultiple to prevent the DAG orchestration from displaying its graphics. It seems to be working and they said a full fix would be implemented soon.
spark.conf.set("spark.notebookutils.run.progressbar.enabled", "false")
Instead of reusing the same session for every runMultiple batch, try Create a new Livy session per DAG chunk. This avoids log cursor conflicts and buffer overflow across chunks.
Did you have any specific detail for this? I believe the livy session and the notebook session can't be seperated. When I forced a restart of the livy session the notebook stopped.