Forum Discussion

PAVAN_111's avatar
PAVAN_111
New Member
10 months ago
Solved

Microsoft Fabric – Event trigger fires multiple times when multiple files arrive

Hi Team, I am using Microsoft Fabric event-based triggers with pipelines. When multiple files arrive at the same time, Fabric creates one event per file, so the pipeline is triggered multiple times...
data engineering
  • deborshi_nag's avatar
    deborshi_nag
    10 months ago

    Hello PAVAN_111 

     

    Native “group/aggregate events into one trigger” isn’t available today. Fabric’s event-based triggers (via Real‑Time hub/Activator) create a separate event—and therefore a separate pipeline run—for each file event that matches your filter. 
    Best‑practice is to design for batching inside your pipeline (or via a small orchestrator pattern) so only one run does the work and any concurrently triggered runs quickly no‑op.
     
    You can use the following pattern:
     
    Allow event triggers to fire, but ensure only one pipeline run actually processes the batch. Others detect the lock and exit fast.
     

    How to implement:

    1. In Pipeline settings, set Concurrency to 1 so runs queue; only one run is active. (This setting is available in ADF/Synapse and behaves similarly in Fabric.) 
    2. At the start of the pipeline:
      • Try to acquire a lease/lock, e.g., write a lock file in a control folder or set a flag row in a control table (Lakehouse or SQL). If a lock exists → skip (return success/no‑op).
    3. Once locked:
      • Wait briefly (e.g., 30–120 seconds) to allow the full file burst to land.
      • List files in the inbound folder, build a batch, and process them together.
      • Move/archive processed files (so subsequent queued runs see “nothing new” and exit).
    4. Release the lock.

    Here's the control flow you can use in your Fabric pipeline:

     

    • Get/Set Lock

      • If Condition: lock_exists() → true → Set variable “SkippedDueToLock” = true → End
      • Else → Create lock (e.g., write /control/ingest.lock or INSERT INTO control_batch(status='running'))

    • Wait (optional)

      • Wait activity: 60–120 seconds (tune to your source uploader behavior)

    • Enumerate batch

      • Get Metadata/List → enumerate /raw/inbound/YYYY/MM/DD
      • Filter to only new files (since last watermark or not in archive)

    • Process

      • ForEach files → copy/transform (or load to staging and Spark job to batch merge)

    • Archive + Watermark

      • Move files to /raw/archive/...
      • Update control table last_processed_timestamp / last_processed_file

    • Release lock

      • Delete /control/ingest.lock or set control row status='completed'

     If this helps please mark this as a solution or a thumbs up! 

Anonymous's avatar
Anonymous
Not applicable
9 months ago

Hi PAVAN_111 ,

There is one option you can try. While creating the event based trigger, in eventstream group the events for the files using group by option , add appropriate sliding window as per your usecase. This will reduce the number of events generation which are eventually passed to activator. 
Now the key the setup would the groupby operation and the sliding window. 

Let me know if you have any doubts !!