Forum Discussion

imtiazali's avatar
imtiazali
Regular Visitor
8 months ago
Solved

Microsoft Fabric Prevent multiple pipeline triggers when multiple files arrive simultaneously.

Hi Team,   I am working with Microsoft Fabric event-based triggers in a data pipeline scenario.   Currently, when multiple files arrive at the same time in the source location, Fabric creates one...
data engineering
  • nielsvdc's avatar
    nielsvdc
    8 months ago

    Hi imtiazali, currently in Fabric event-based triggers are per event. A run is created for each file event that matches your filter. There isn’t a native "group multiple file-created events into one" trigger.

     

    Here are some options you can consider.

     

    1. Schedule trigger every x minutes
      • Maybe not what you are looking for, but you can schedule the pipeline to run every 1-5 minutes between a given time frame and process all the available files. If the processing of files takes longer than the interval, move the files to a processing folder first to ensure idempotency.
      • The trade-off is that you will still see runs for all files, but that do nothing, because there are now files left to process.
    2. Concurrency = 1+"first-file" guard
      • Leave the event trigger enabled, set pipeline concurrency to 1 so only one run executes at a time. All other trigger events are queued and processed after the first run.
      • In the pipeline implement a guard/lease with a check a lock file or control table.
        • If no active batch, acquire the lock, list all new files, process them together, release the lock.
        • If another run is already active (or just completed the batch), skip and succeed.
      • Multiple triggers can fire, but only one run actually performs work; others become quick no‑ops.
      • Optionally, add a Wait activity to the pipeline for waiting until all files have been written to location that you are monitoring.
      • The trade-off is that you will still see runs for all files, but that do nothing, because there are now files left to process.
    3. Use Data Activitor for files on OneLake/Blob?
      • I haven't played around with this option yet and I currently don't have time to try it, but maybe it could work.
      • Configure OneLake/Blob events in Data Activator to start a pipeline when an event happens. For Blob storage you might need to configure Azure Event Hub.
      • Configure the Wait time for late-arriving events option to wait for all the data to be available. Play around with finding the correct condition.

    Hope this helps. If so, please give kudos ‌‌👍 and mark as Accepted Solution ‌‌✔️ to help others. If you resolved your question, let us know what worked for you.

  • deborshi_nag's avatar
    deborshi_nag
    8 months ago

    Hi imtiazali 

     

    Native “group/aggregate events into one trigger” isn’t available today. Fabric’s event-based triggers (via Real‑Time hub/Activator) create a separate event—and therefore a separate pipeline run—for each file event that matches your filter. 
    Best‑practice is to design for batching inside your pipeline (or via a small orchestrator pattern) so only one run does the work and any concurrently triggered runs quickly no‑op.
     
    You can use the following pattern:
     
    Allow event triggers to fire, but ensure only one pipeline run actually processes the batch. Others detect the lock and exit fast.
     

    How to implement:

    1. In Pipeline settings, set Concurrency to 1 so runs queue; only one run is active. (This setting is available in ADF/Synapse and behaves similarly in Fabric.) 
    2. At the start of the pipeline:
      • Try to acquire a lease/lock, e.g., write a lock file in a control folder or set a flag row in a control table (Lakehouse or SQL). If a lock exists → skip (return success/no‑op).
    3. Once locked:
      • Wait briefly (e.g., 30–120 seconds) to allow the full file burst to land.
      • List files in the inbound folder, build a batch, and process them together.
      • Move/archive processed files (so subsequent queued runs see “nothing new” and exit).
    4. Release the lock.

    Here's the control flow you can use in your Fabric pipeline:

     

    • Get/Set Lock

      • If Condition: lock_exists() → true → Set variable “SkippedDueToLock” = true → End
      • Else → Create lock (e.g., write /control/ingest.lock or INSERT INTO control_batch(status='running'))

    • Wait (optional)

      • Wait activity: 60–120 seconds (tune to your source uploader behavior)

    • Enumerate batch

      • Get Metadata/List → enumerate /raw/inbound/YYYY/MM/DD
      • Filter to only new files (since last watermark or not in archive)

    • Process

      • ForEach files → copy/transform (or load to staging and Spark job to batch merge)

    • Archive + Watermark

      • Move files to /raw/archive/...
      • Update control table last_processed_timestamp / last_processed_file

    • Release lock

      • Delete /control/ingest.lock or set control row status='completed'

     If this helps please mark this as a solution. 

v-hashadapu's avatar
v-hashadapu
Icon for Community Support rankCommunity Support
8 months ago

Hi imtiazali , Thank you for reaching out to the Microsoft Community Forum.

 

We find the answers shared by deborshi_nag & nielsvdc are appropriate. Can you please confirm if they worked for you. It will help others with similar issues find the answer easily.

 

Thank you deborshi_nag & nielsvdc  for your valuable responses.