Forum Discussion
Spark Structured Streaming Continuous mode
Hi smpa01 , Yes, for now that is the case. but hoping it will change soon.
If this helped solve the issue, please consider marking it 'Accept as Solution' so others with similar queries may find it more easily. If not, please share the details, always happy to help.
Thank you.
Hi smpa01 , Thank you for reaching out to the Microsoft Community Forum.
Microsoft Fabric doesn’t support Continuous mode for notebooks like Databricks does. Instead, Fabric notebooks run in batch mode, each scheduled run starts a Spark session, executes the code and then shuts down, which stops any streaming queries. This means Fabric can’t natively support always-on, continuous streaming jobs out of the box.
However, you can achieve near-continuous streaming using Spark Structured Streaming with checkpointing and frequent scheduling. To do this, write your streaming logic to include a checkpointLocation, which allows Spark to persist the query’s state and resume from where it left off in the next run. Set a short trigger interval, such as 10 seconds, so the query processes small batches of data frequently during each run.
To keep the notebook active long enough for meaningful processing, use a time.sleep() function, typically for about 5 minutes. This gives your streaming query time to process incoming data before the Spark session ends. Finally, schedule the notebook to run every 5 to 10 minutes. When combined with checkpointing, this setup ensures that each run continues smoothly from the last, minimizing data gaps. Expect around 5–10 minutes of latency per cycle, plus 30–60 seconds for cluster spin-up time.
If this helped solve the issue, please consider marking it 'Accept as Solution' so others with similar queries may find it more easily. If not, please share the details, always happy to help.
Thank you.
v-hashadapu Thanks for the explanation and response.
With batch streaming, the streaming would probably work as desired for immutable data.
But batch straming would not work for a scenario when source data mutates and the goal is to have any mutation streamed downstream whatsoever.
So at current capabilties, true streaming is not possible for any use case scenarios whatsoever as it is in databricks, fair to conclude that?
Hi smpa01 , Yes, for now that is the case. but hoping it will change soon.
If this helped solve the issue, please consider marking it 'Accept as Solution' so others with similar queries may find it more easily. If not, please share the details, always happy to help.
Thank you.
