Azure Data Factory Pipeline Performance Degradation

[MrTechie]

Hello, community, 

I am experiencing a performance issue with an event-based pipeline that runs multiple times a day. Under normal conditions, when fewer than ten pipeline runs execute in parallel, each run completes in approximately 7 minutes. However, when more parallel runs occur, the execution time for each pipeline extends to over an hour.

Despite this increase in execution time, the actual data loading into SQL is completing within 3 minutes, indicating that SQL is not the bottleneck. But the pipeline duration shows an average of 1.5 hours. The cause of this extended execution time is unclear, and I want to understand what might be contributing to the delay when more parallel runs are triggered.

In this case, around 40 parallel pipeline runs are occurring.

Note: I didn't set up any concurrency in the pipeline settings. (The box is empty)

Integration Run: Auto Resolve Integration Run time.

SQL: 400 DTUs (Reaching max 80% in busy time)

Below are some of the runs:

I suspect this is giving us a large bill at the end of the month.

Please give me some suggestions to optimize this duration. 

TIA

[Anonymous · Accepted solution]

Hi @MrTechie ,

Based on the description, the SQL data takes only 3 minutes to load, but the total pipeline time is 1.5 hours, suggesting that bottlenecks may be occurring in the following areas: IR resource contention, queuing of activities, or dependency waiting.
Check the CPU/Memory utilization of the Integration Runtime in Azure Monitor.  Or check the activity-level time distribution in Pipeline Run Details to identify if any activities have been in the Queued or Ready state for a long time.

For more details, you can refer to below document:

How to monitor pipeline runs - Microsoft Fabric | Microsoft Learn

Integration runtime - Azure Data Factory & Azure Synapse | Microsoft Learn

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[nilendraFabric]

Hello @MrTechie 

one thing which is notticable here is that all the delayed run are before 12:00 am.

Here are few observations :

While no explicit concurrency settings are configured, ADF pipelines by default allow multiple parallel executions. However, excessive parallelism can lead to resource saturation at the source, sink, or IR level.

When many pipelines run in parallel, ADF may queue activities or experience delays in starting new clusters for data flows or copy activities.

The Auto-Resolve Integration Runtime (IR) may not be scaling efficiently to handle the increased number of parallel pipeline runs. 

Here are mumy suggestions:

Create a custom IR with higher compute capacity tailored to your workload. Adjust the core count and memory size based on the expected parallelism.
• Enable Time-to-Live (TTL): Use TTL settings for IR to reduce cluster startup time between pipeline executions

Configure concurrency settings at the pipeline level to limit the number of simultaneous runs. This prevents overloading resources

See if this helps