How to orchestrate streaming pipelines

Hi innocence84 ,
Great follow-up questions!
Is clicking Run All enough for keeping the notebook session alive?
Clicking "Run All" will start the notebook execution, but it does not guarantee that the notebook session will stay alive indefinitely. Spark Structured Streaming requires a continuously active session to keep ingesting data.

To keep the session alive:
Do not close the notebook or browser tab.
Make sure your Microsoft Fabric capacity is set to stay on (not auto-pause), so the Spark session doesn't terminate due to inactivity or capacity shutdown.
Alternatively, use a Fabric Pipeline with a trigger to re-launch the notebook automatically, although this may cause some latency during cluster spin-up unless the capacity is always-on.
So, while "Run All" starts the job, the session must be kept open and the Spark cluster active for it to keep running as a true long-running job.
What is always-on execution model?
The always-on execution model means that the compute resources (Spark capacity) are continuously running, so notebooks and streaming jobs can execute without delays caused by cluster spin-up or cold starts.
In Microsoft Fabric, this is typically achieved by:
Keeping the capacity always-on under the Fabric settings (i.e., prevent auto-pause).
Using Fabric Pipelines to orchestrate jobs in a way that aligns with this model (e.g., triggering notebooks as soon as new data arrives or at regular intervals without waiting for cluster startup).
This model is crucial for low-latency streaming scenarios, where immediate data processing is required without downtime or lag due to cluster initialization.
Ingest, filter, and transform real-time events and send them to a Microsoft Fabric lakehouse - Microsoft Fabric | Microsoft Learn

If this post helped resolve your issue, please consider giving it Kudos and marking it as the Accepted Solution. This not only acknowledges the support provided but also helps other community members find relevant solutions more easily.
We appreciate your engagement and thank you for being an active part of the community.

Best regards,
LakshmiNarayana.