Forum Discussion

GlassShark1's avatar
GlassShark1
Icon for Helper III rankHelper III
1 year ago
Solved

Can't ingest via API/Eventsteam - Notebook hanging

Starting from scratch with RTA and struggling. I've created an Eventhouse, a Notebook (that calls an API and returns json) and an Eventstream. The notebook is scheduled every 5 minutes. The E...
eventstream
  • GlassShark1's avatar
    GlassShark1
    1 year ago

    eventually took a different approach to this to solve.

    tried a longer refresh schedule and jobs still ended up hanging. Used azure bus instead of the event hub producer (which i think was having trouble closing maybe) and set the whole script in a continous loop with a timer spacing the polling as opposed managing a refresh schedule in the notebook itself.

lbendlin's avatar
lbendlin
Icon for Super User rankSuper User
1 year ago
The notebook is scheduled every 5 minutes.

5 minutes seems to be the sweet/sour spot where it is not economical yet to keep the spark session running, and where the spin-up delay of the new session has a considerable impact.  Most likely your environment is busy spinning up and tearing down, and not doing much else.