Unexpected Pipeline Slowness / Increase in Data Flow vCore Consumption

[AngusWhite]

Hi,

We have been running the same pipelines on a weekly basis and have seen consistent results when it comes to run time / vCore consumption / cost. 

A week ago however, we noticed that our pipelines have been running extremely slowly, and as a result have consumed a huge amount of V-core processing. 

All our pipelines are impacted, but is a single example:

Pipeline duration:

07/14/2024 - 30s
07/20/2024 - 10m 6s

Data flow vCore-hour:
07/14/2024 - 0.0722
07/20/2024 - 2.614

Any ideas on what is causing the impact to our pipeline performance? We have not changed  any run time variables, and the number of rows / records being processed has remained the same.

One theory we have is that the Azure resource / VM associated with the run time was low on space, would this have an impact?
Thanks,

Angus

[AngusWhite]

Hi Nono, 

Many thanks for getting back to me. 

We are actually reffering to the Azure environment rather than fabric (perhpas I've posted in the wrong forum).

You mentioned more data flows / data volumes, however nothing has changed from a volume perspective. We've made no changes to the pipelines. We are still processesing the same number of records. 

It's a bit of a mystery - any insight you could provide would be much appreciated.

Thanks,

Angus

[Anonymous · Accepted solution]

Hi @AngusWhite 

Thank you very much for your question!

This community is dedicated to discussing issues related to Microsoft Fabric.

To provide you with more professional help, we recommend using a dedicated Microsoft Q&A forum for Azure Data Factory (ADF) related queries.

They are always ready to help you better. Here's a link to your question in Microsoft Questions:

Ask a question - Microsoft Q&A

Regards,

Nono Chen

[Anonymous]

Hi @AngusWhite 

"One theory we have is that the Azure resource / VM associated with the run time was low on space, would this have an impact?" That could be one reason.

Based on the screenshots you provided, I noticed that the number of dataflow seems to have increased.

There are many factors that can change the amount of data, such as:

Data source changes. If the data source changes, such as the addition of a new data table or data set, it may result in an increase in the number of data streams.

Business requirements change. As business requirements change, more data flows may need to be processed.

Data processing logic changes. If the data processing logic changes, such as adding a new data processing step or branch, the number of data streams may increase.

Data partitioning policy. Adjustments to the data partitioning policy may result in an increase in the number of data streams.

The amount of data increases. Over time, an increase in the amount of data may lead to an increase in the number of data streams.

System optimization. Sometimes in order to optimize system performance, a large data stream may be split into multiple smaller data streams for better management and processing.

Each data stream consumes a certain amount of computing resources. As the number of data streams increases, so does resource consumption, potentially leading to resource contention and performance degradation.

You can use the Microsoft Fabric Capacity Metrics application to monitor resource usage. Designed for monitoring Microsoft Fabric capacity, this application provides detailed capacity consumption information to help you make decisions about capacity resource usage.

What is the Microsoft Fabric Capacity Metrics app? - Microsoft Fabric | Microsoft Learn

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.