Forum Discussion
Azure Data Factory Pipeline Performance Degradation
Hi MrTechie ,
Based on the description, the SQL data takes only 3 minutes to load, but the total pipeline time is 1.5 hours, suggesting that bottlenecks may be occurring in the following areas: IR resource contention, queuing of activities, or dependency waiting.
Check the CPU/Memory utilization of the Integration Runtime in Azure Monitor. Or check the activity-level time distribution in Pipeline Run Details to identify if any activities have been in the Queued or Ready state for a long time.For more details, you can refer to below document:
How to monitor pipeline runs - Microsoft Fabric | Microsoft Learn
Integration runtime - Azure Data Factory & Azure Synapse | Microsoft Learn
Best Regards,
Adamk KongIf this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Hello MrTechie
one thing which is notticable here is that all the delayed run are before 12:00 am.
Here are few observations :
While no explicit concurrency settings are configured, ADF pipelines by default allow multiple parallel executions. However, excessive parallelism can lead to resource saturation at the source, sink, or IR level.
When many pipelines run in parallel, ADF may queue activities or experience delays in starting new clusters for data flows or copy activities.
The Auto-Resolve Integration Runtime (IR) may not be scaling efficiently to handle the increased number of parallel pipeline runs.
Here are mumy suggestions:
Create a custom IR with higher compute capacity tailored to your workload. Adjust the core count and memory size based on the expected parallelism.
• Enable Time-to-Live (TTL): Use TTL settings for IR to reduce cluster startup time between pipeline executions
Configure concurrency settings at the pipeline level to limit the number of simultaneous runs. This prevents overloading resources
See if this helps