Forum Discussion

zakeer1517's avatar
zakeer1517
Regular Visitor
1 year ago
Solved

InvalidHttpRequestToLivy: from cannot be less than 0 HTTP status code: 400

We are currently integrating data from the bronze to silver layer using a DAG that runs notebooks in parallel with a concurrency level of 5. However, the process consistently fails after approximatel...
data engineering
  • zakeer1517's avatar
    zakeer1517
    1 year ago

    Hi v-kpoloju-msft 

     

    I haven't opened any support ticket. I was able to fix the issue by increasing the concurrency limit to 20 by which the execution time falls with in the session time out limit (28-29 mins). 

     

    Thanks for the support. 

     

     

Vinodh247's avatar
Vinodh247
Super User
1 year ago

The error from cannot be less than 0 hints at an internal pagination or offset issue in Livy's REST API used by notebookutils.notebook.runMultiple. After prolonged concurrent polling (especially under parallel loads), it might be attempting an invalid request with a negative offset.

 

Why it shows up after 28/ 29 mins consistently? Read the following possible reasons.

  • There could be an internal TTL or token/session refresh window in Livy or Spark (ex: 30min inactivity or API paging expiration?).
  • Fabric's notebook orchestrator may be batching or chunking execution state (like polling logs/outputs), and running into a paging math bug.

You are using a concurrency of 5 for 172 notebooks, there are chances that it can create...

  • Longlived sessions lingering in memory (especially if each notebook creates new Spark sessions instead of reusing).
  • Driver memory or thread pool exhaustion, especially if Fabric’s backend queues and polls results via Livy and keeps buffers in memory.

Even though you scaled to F8, this does not change concurrency queues or polling behavior in Livy unless explicitly managed.

 

Common troubleshooting fix that you can try: 

  • Instead of firing all 172 in one DAG, batch it into chunks. Try two DAG runs of 86 tables each, or
  • Use concurrency 3 with fewer total DAG nodes per batch. This avoids overwhelming the Livy poller or execution state tracker.
  • If possible, wrap notebook.runMultiple in a retry logic to catch and reinitiate after a failure.
  • Use monitoring in Fabric to check Livy session limits, queued job durations, and driver memory usage.
  • Look for patterns like exactly 30 min duration, queued notebook executions, or polling failures.

 

 

Please 'Kudos' and 'Accept as Solution' if this answered your query.