Spark Notebook termination after ~14 days (Eventstream Destination)

[Hamidr]

I have a realtime ingestion process as the following :

• In an eventstream ingesting events from an eventhub
• Doing some operations (filter events , ...)
• Set a notebook as destination of the eventstream
• In the notebook : apply upsert operation and append or updata data into a Lakehouse table

when I add the notebook as destination an instance of notebook starts to run (I can see the activity in monitor) and working fine (the events are ingested and in Lakehouse table, evrything looks great) but after almost 14 days the notebook suddenly stops and I need to manually delete the notebook from eventstrteam and add it again as destination.
Does it seem the native behaviour of Fabric to stop spending resources or there is a setting I can set it to let it run forever?
Thanks

[Hamidr]

@rizalard0684 Thanks for your solution , I did implment same at the beginning but changed the solution to have  more near real time approach but it seems what I did before and you also suggested is the only solution for now.

[v-ssriganesh]

Hi @Hamidr,

Thank you for posting your query in the Microsoft Fabric Community Forum, and thanks to @rizalard0684 for sharing valuable insights.

Could you please confirm if your query has been resolved by the provided solutions? This would be helpful for other members who may encounter similar issues.

Thank you for being part of the Microsoft Fabric Community.

[rizalard0684 · Accepted solution]

Hey @Hamidr  I believe this is an expected behavior today in Microsoft Fabric.

When you set a Notebook as an Eventstream destination, Fabric spins up a managed Spark job.
That job runs continuously, consumes capacity and has an internal lifetime limit.

After around 14 days, Fabric terminates the job automatically (this is a platform safeguard).

At the moment there is no setting to make it run indefinitely.

I'd recommend you do lightweight transformations in Eventstream, then land data to Lakehouse (append) or Eventhouse / KQL database (as KQL table).

Then add scheduled or triggered notebooks to do upsert / merge logic for downstream consumptions.

Appreciate if you can 'Kudos' and/or 'Accept as Solution' if this answered your query.