Forum Discussion
Azure Data Factory Pipeline Performance Degradation
- Anonymous1 year ago
Hi MrTechie ,
Based on the description, the SQL data takes only 3 minutes to load, but the total pipeline time is 1.5 hours, suggesting that bottlenecks may be occurring in the following areas: IR resource contention, queuing of activities, or dependency waiting.
Check the CPU/Memory utilization of the Integration Runtime in Azure Monitor. Or check the activity-level time distribution in Pipeline Run Details to identify if any activities have been in the Queued or Ready state for a long time.For more details, you can refer to below document:
How to monitor pipeline runs - Microsoft Fabric | Microsoft Learn
Integration runtime - Azure Data Factory & Azure Synapse | Microsoft Learn
Best Regards,
Adamk KongIf this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
Hi MrTechie ,
Based on the description, the SQL data takes only 3 minutes to load, but the total pipeline time is 1.5 hours, suggesting that bottlenecks may be occurring in the following areas: IR resource contention, queuing of activities, or dependency waiting.
Check the CPU/Memory utilization of the Integration Runtime in Azure Monitor. Or check the activity-level time distribution in Pipeline Run Details to identify if any activities have been in the Queued or Ready state for a long time.
For more details, you can refer to below document:
How to monitor pipeline runs - Microsoft Fabric | Microsoft Learn
Integration runtime - Azure Data Factory & Azure Synapse | Microsoft Learn
Best Regards,
Adamk Kong
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.