Forum Discussion

Hamidr's avatar
Hamidr
Frequent Visitor
4 months ago
Solved

Spark Notebook termination after ~14 days (Eventstream Destination)

I have a realtime ingestion process as the following : In an eventstream ingesting events from an eventhub Doing some operations (filter events , ...) Set a notebook as destination of the eventst...
Eventstream
  • rizalard0684's avatar
    rizalard0684
    4 months ago

    Hey Hamidr  I believe this is an expected behavior today in Microsoft Fabric.

    When you set a Notebook as an Eventstream destination, Fabric spins up a managed Spark job.
    That job runs continuously, consumes capacity and has an internal lifetime limit.

    After around 14 days, Fabric terminates the job automatically (this is a platform safeguard).

    At the moment there is no setting to make it run indefinitely.

     

    I'd recommend you do lightweight transformations in Eventstream, then land data to Lakehouse (append) or Eventhouse / KQL database (as KQL table).

    Then add scheduled or triggered notebooks to do upsert / merge logic for downstream consumptions.

     

    Appreciate if you can 'Kudos' and/or 'Accept as Solution' if this answered your query.

v-ssriganesh's avatar
v-ssriganesh
Community Support
3 months ago

Hi Hamidr,

Thank you for posting your query in the Microsoft Fabric Community Forum, and thanks to rizalard0684 for sharing valuable insights.

 

Could you please confirm if your query has been resolved by the provided solutions? This would be helpful for other members who may encounter similar issues.

 

Thank you for being part of the Microsoft Fabric Community.