Forum Discussion
How to orchestrate streaming pipelines
Hi innocence84 ,
Great follow-up questions!
Is clicking Run All enough for keeping the notebook session alive?
Clicking "Run All" will start the notebook execution, but it does not guarantee that the notebook session will stay alive indefinitely. Spark Structured Streaming requires a continuously active session to keep ingesting data.To keep the session alive:
Do not close the notebook or browser tab.
Make sure your Microsoft Fabric capacity is set to stay on (not auto-pause), so the Spark session doesn't terminate due to inactivity or capacity shutdown.
Alternatively, use a Fabric Pipeline with a trigger to re-launch the notebook automatically, although this may cause some latency during cluster spin-up unless the capacity is always-on.
So, while "Run All" starts the job, the session must be kept open and the Spark cluster active for it to keep running as a true long-running job.
What is always-on execution model?
The always-on execution model means that the compute resources (Spark capacity) are continuously running, so notebooks and streaming jobs can execute without delays caused by cluster spin-up or cold starts.
In Microsoft Fabric, this is typically achieved by:
Keeping the capacity always-on under the Fabric settings (i.e., prevent auto-pause).
Using Fabric Pipelines to orchestrate jobs in a way that aligns with this model (e.g., triggering notebooks as soon as new data arrives or at regular intervals without waiting for cluster startup).
This model is crucial for low-latency streaming scenarios, where immediate data processing is required without downtime or lag due to cluster initialization.
Ingest, filter, and transform real-time events and send them to a Microsoft Fabric lakehouse - Microsoft Fabric | Microsoft LearnIf this post helped resolve your issue, please consider giving it Kudos and marking it as the Accepted Solution. This not only acknowledges the support provided but also helps other community members find relevant solutions more easily.
We appreciate your engagement and thank you for being an active part of the community.Best regards,
LakshmiNarayana.
Hi innocence84 ,
Great question!
To run Notebook 1 as a long-running Spark Structured Streaming job in Microsoft Fabric, follow these steps:
Where to run Notebook 1:
Run it inside the Microsoft Fabric Lakehouse environment, specifically in a Lakehouse notebook with Spark runtime.
Fabric supports Spark-based notebooks for streaming workloads using Delta Lake and Event Hub integration.
How to run it as a long-running job:
Create a Lakehouse Notebook:
In Microsoft Fabric, go to your Lakehouse.
Create or open Notebook 1.
Write your structured streaming code to read from Azure Event Hub and write to Delta Lake (bronze layer).
Use Spark Structured Streaming API in PySpark:
Example:
df = (
spark.readStream
.format("eventhubs")
.option("eventhubs.connectionString", "<your_connection_string>")
.load()
)
df.writeStream \
.format("delta") \
.option("checkpointLocation", "Tables/bronze/_checkpoints/") \
.start("Tables/bronze/")
Set the notebook to run continuously:
When starting the notebook, choose "Run All" to execute the code.
Keep the notebook session alive (do not rely on scheduled runs).
Alternatively, you can run this notebook from a Microsoft Fabric Pipeline with a trigger but ensure cluster warm-up is minimized (e.g., keep capacity always-on).
Use Fabric Pipelines for orchestration (optional):
You can orchestrate Notebook 1 using Fabric Pipelines with a trigger-based or always-on execution model.
This ensures the job runs without manual intervention and maintains a low-latency ingestion pipeline.
If this post helped resolve your issue, please consider giving it Kudos and marking it as the Accepted Solution. This not only acknowledges the support provided but also helps other community members find relevant solutions more easily.
We appreciate your engagement and thank you for being an active part of the community.
Best Regards,
Lakshmi Narayana.
thank you,
for the solution 1, is clicking run all enough for Keep the notebook session alive ?
for the solution 2, what is always-on execution model.?
