Forum Discussion
Power ParticipantWhat are the scenarios for Delta Streaming ?
Hi DennesTorres,
Thank you for your patience and for sharing your thoughts!
After reviewing the link you provided, Richbenmintz offered a comprehensive overview of Delta Streaming's capability to operate continuously. Their points about the persistence of the streaming job, even if the notebook session is stopped, are accurate. Streaming jobs can indeed continue running independently of the notebook session, enabling near real-time analytics.
Providing comprehensive insights and detailed analysis:
So, my response aimed to clarify the behavior of streaming in notebooks, but it may have caused some confusion. While it's true that if you don’t call awaitTermination() or stop(), the stream can keep running, I should have emphasized that the stream will indeed stop if the notebook session is terminated.- Yes, Delta Streaming can be continuous, but it typically requires a Spark job definition for long-term execution.
- The streaming job persists independently of the notebook session, but if the session is stopped, the stream will stop as well.
- You can manage streaming jobs through the Spark UI or programmatically, but they will need to be restarted if the session ends.
I appreciate your willingness to experiment further, as hands-on experience is invaluable for understanding these concepts. If this proves helpful, kindly Accept it as a solution and leave a "Kudos" so other members can easily find it.
Thank you.
Power ParticipantHi,
"
- Stopping the notebook session doesn't terminate the streaming job. The session acts as a trigger for initiating the stream, but the underlying Spark job continues processing data.
- Spark provides tools to manage streaming jobs. You can monitor, stop, and restart them using the Spark UI or programmatically."
Could you provide details about how to do this, how to manage streaming jobs after the session finished ?
Community SupportHi DennesTorres ,
That's a great follow-up question! Here's how you can manage Delta Streaming jobs after your notebook session finishes:
- Once your Delta Streaming job is initiated through the notebook, locate the link to the Spark UI in your notebook's output. This UI provides a web interface to monitor and manage Spark applications, including your streaming job. Within the Spark UI, look for the Streaming tab or section. This will display a list of active streaming queries, including your Delta Streaming job.
- Each streaming job entry will provide details like its name, ID, and current status. You can then perform actions like:
- Stop: Clicking the stop button associated with your job will gracefully terminate the streaming query.
- Monitor Progress: You can monitor the progress of your streaming job, including the number of records processed, latency, and any errors encountered.
- Alternatively, you can leverage the Spark Session API within your notebook to programmatically manage streaming jobs. Here's a basic example:
Code:
from pyspark.sql import SparkSession
spark = spark.sparkContext.getOrCreate()._jsparkSession
spark.streams.active.stop()
streaming_query = spark.streams.getQueryByName("your_query_name")
streaming_query.stop()
By following these methods, you can effectively manage your Delta Streaming jobs even after your notebook session has ended. If you find this helpful, please Accept it as a solution and consider leaving a "Kudos" so other members can locate it more easily.
Thank you.