Forum Discussion

PAVAN_111's avatar
PAVAN_111
New Member
10 months ago
Solved

Microsoft Fabric – Event trigger fires multiple times when multiple files arrive

Hi Team, I am using Microsoft Fabric event-based triggers with pipelines. When multiple files arrive at the same time, Fabric creates one event per file, so the pipeline is triggered multiple times...
data engineering
  • deborshi_nag's avatar
    deborshi_nag
    10 months ago

    Hello PAVAN_111 

     

    Native “group/aggregate events into one trigger” isn’t available today. Fabric’s event-based triggers (via Real‑Time hub/Activator) create a separate event—and therefore a separate pipeline run—for each file event that matches your filter. 
    Best‑practice is to design for batching inside your pipeline (or via a small orchestrator pattern) so only one run does the work and any concurrently triggered runs quickly no‑op.
     
    You can use the following pattern:
     
    Allow event triggers to fire, but ensure only one pipeline run actually processes the batch. Others detect the lock and exit fast.
     

    How to implement:

    1. In Pipeline settings, set Concurrency to 1 so runs queue; only one run is active. (This setting is available in ADF/Synapse and behaves similarly in Fabric.) 
    2. At the start of the pipeline:
      • Try to acquire a lease/lock, e.g., write a lock file in a control folder or set a flag row in a control table (Lakehouse or SQL). If a lock exists → skip (return success/no‑op).
    3. Once locked:
      • Wait briefly (e.g., 30–120 seconds) to allow the full file burst to land.
      • List files in the inbound folder, build a batch, and process them together.
      • Move/archive processed files (so subsequent queued runs see “nothing new” and exit).
    4. Release the lock.

    Here's the control flow you can use in your Fabric pipeline:

     

    • Get/Set Lock

      • If Condition: lock_exists() → true → Set variable “SkippedDueToLock” = true → End
      • Else → Create lock (e.g., write /control/ingest.lock or INSERT INTO control_batch(status='running'))

    • Wait (optional)

      • Wait activity: 60–120 seconds (tune to your source uploader behavior)

    • Enumerate batch

      • Get Metadata/List → enumerate /raw/inbound/YYYY/MM/DD
      • Filter to only new files (since last watermark or not in archive)

    • Process

      • ForEach files → copy/transform (or load to staging and Spark job to batch merge)

    • Archive + Watermark

      • Move files to /raw/archive/...
      • Update control table last_processed_timestamp / last_processed_file

    • Release lock

      • Delete /control/ingest.lock or set control row status='completed'

     If this helps please mark this as a solution or a thumbs up! 

deborshi_nag's avatar
deborshi_nag
Icon for Super User rankSuper User
10 months ago

Hello PAVAN_111 

 

Native “group/aggregate events into one trigger” isn’t available today. Fabric’s event-based triggers (via Real‑Time hub/Activator) create a separate event—and therefore a separate pipeline run—for each file event that matches your filter. 
Best‑practice is to design for batching inside your pipeline (or via a small orchestrator pattern) so only one run does the work and any concurrently triggered runs quickly no‑op.
 
You can use the following pattern:
 
Allow event triggers to fire, but ensure only one pipeline run actually processes the batch. Others detect the lock and exit fast.
 

How to implement:

  1. In Pipeline settings, set Concurrency to 1 so runs queue; only one run is active. (This setting is available in ADF/Synapse and behaves similarly in Fabric.) 
  2. At the start of the pipeline:
    • Try to acquire a lease/lock, e.g., write a lock file in a control folder or set a flag row in a control table (Lakehouse or SQL). If a lock exists → skip (return success/no‑op).
  3. Once locked:
    • Wait briefly (e.g., 30–120 seconds) to allow the full file burst to land.
    • List files in the inbound folder, build a batch, and process them together.
    • Move/archive processed files (so subsequent queued runs see “nothing new” and exit).
  4. Release the lock.

Here's the control flow you can use in your Fabric pipeline:

 

  • Get/Set Lock

    • If Condition: lock_exists() → true → Set variable “SkippedDueToLock” = true → End
    • Else → Create lock (e.g., write /control/ingest.lock or INSERT INTO control_batch(status='running'))

  • Wait (optional)

    • Wait activity: 60–120 seconds (tune to your source uploader behavior)

  • Enumerate batch

    • Get Metadata/List → enumerate /raw/inbound/YYYY/MM/DD
    • Filter to only new files (since last watermark or not in archive)

  • Process

    • ForEach files → copy/transform (or load to staging and Spark job to batch merge)

  • Archive + Watermark

    • Move files to /raw/archive/...
    • Update control table last_processed_timestamp / last_processed_file

  • Release lock

    • Delete /control/ingest.lock or set control row status='completed'

 If this helps please mark this as a solution or a thumbs up! 

v-menakakota's avatar
v-menakakota
Icon for Community Support rankCommunity Support
10 months ago

Hi PAVAN_111 
Thanks for reaching out to the Microsoft fabric community forum. 

 

I would also take a moment to thank  deborshi_nag , tayloramy  and tayloramy  , for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.

I hope the above details help you fix the issue. If you still have any questions or need more help, feel free to reach out. We’re always here to support you .

 

 

Best Regards, 
Community Support Team  

  • v-menakakota's avatar
    v-menakakota
    Icon for Community Support rankCommunity Support
    9 months ago

    Hi PAVAN_111 ,

    I hope the above details help you fix the issue. If you still have any questions or need more help, feel free to reach out. We’re always here to support you .

     

    Best Regards, 
    Community Support Team