Forum Discussion

Hamidr's avatar
Hamidr
Frequent Visitor
4 months ago
Solved

Spark Notebook termination after ~14 days (Eventstream Destination)

I have a realtime ingestion process as the following : In an eventstream ingesting events from an eventhub Doing some operations (filter events , ...) Set a notebook as destination of the eventst...
  • rizalard0684's avatar
    4 months ago

    Hey Hamidr  I believe this is an expected behavior today in Microsoft Fabric.

    When you set a Notebook as an Eventstream destination, Fabric spins up a managed Spark job.
    That job runs continuously, consumes capacity and has an internal lifetime limit.

    After around 14 days, Fabric terminates the job automatically (this is a platform safeguard).

    At the moment there is no setting to make it run indefinitely.

     

    I'd recommend you do lightweight transformations in Eventstream, then land data to Lakehouse (append) or Eventhouse / KQL database (as KQL table).

    Then add scheduled or triggered notebooks to do upsert / merge logic for downstream consumptions.

     

    Appreciate if you can 'Kudos' and/or 'Accept as Solution' if this answered your query.