Forum Discussion
Spark Notebook termination after ~14 days (Eventstream Destination)
- 4 months ago
Hey Hamidr I believe this is an expected behavior today in Microsoft Fabric.
When you set a Notebook as an Eventstream destination, Fabric spins up a managed Spark job.
That job runs continuously, consumes capacity and has an internal lifetime limit.After around 14 days, Fabric terminates the job automatically (this is a platform safeguard).
At the moment there is no setting to make it run indefinitely.
I'd recommend you do lightweight transformations in Eventstream, then land data to Lakehouse (append) or Eventhouse / KQL database (as KQL table).
Then add scheduled or triggered notebooks to do upsert / merge logic for downstream consumptions.
Appreciate if you can 'Kudos' and/or 'Accept as Solution' if this answered your query.
rizalard0684 Thanks for your solution , I did implment same at the beginning but changed the solution to have more near real time approach but it seems what I did before and you also suggested is the only solution for now.