Data Pipelines-Performance

[AmruthaVarshini]

Hi Team,

Have a pipeline which exports data from ADLS container to SQL Server. Data present in ADLS Container is approximately 30GB of Data. Fabric SKU which we are currently using is F64 SKU capacity. 

Few days before when we are loading data into Sql server using same capacity(same amount of data as well) , it is pretty quick and today when we tried to load same amount of data it took more than expected time.

Would you please provide your insights on why there is a performance difference on same data size and also please provide inputs on change in Performance if data is increased.

[NandanHegde · Accepted solution]

There can be multiple reasons due to change in performance time for the same amount of data :

1) What was the load / active jobs or queries on the SQL server between the 2 instances?

2) Were there any other Fabric jobs being executed between the 2 instances as the capacity is being shared

[v-kongfanf-msft · Accepted solution]

Thanks for the reply from @NandanHegde , please allow me to provide another insight:

Hi @AmruthaVarshini ,

Based on your description, there are a number of factors that can contribute to this condition, and you can troubleshoot from these perspectives as follows:

• Check the CPU, memory, and network bandwidth utilization of the F64 SKU capacity during the pipeline run. If resource utilization is high, it may lead to performance degradation. Tools such as Azure Monitor can be used to check these metrics.
• Check network bandwidth and latency between ADLS and SQL Server. Network issues may cause data transfer to slow down. You can use network monitoring tools to diagnose network performance issues.
• Check if there are other concurrent tasks or pipelines running at the same time, causing competition for resources. Ensure that sufficient resources are available when running critical tasks.

For more details, you can read below document:

Azure Monitor overview - Azure Monitor | Microsoft Learn

Best Regards,
Adamk Kong

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[NandanHegde · Accepted solution]

There can be multiple reasons due to change in performance time for the same amount of data :

1) What was the load / active jobs or queries on the SQL server between the 2 instances?

2) Were there any other Fabric jobs being executed between the 2 instances as the capacity is being shared