topic Re: Performance and execution times in Pipelines

Performance and execution times

PedroCSeq — Tue, 20 Aug 2024 09:52:50 GMT

Hello everyone,

I'm building a lakehouse to serve as the centralized source of truth for an organization and the periodic ingestions are taking a looong time to ingest a relatively small volume of data. The built in fabric activities don't seem to be taking more that they should and the variance on the execution time is low, but the execution of notebooks in the pipelines seems to be broken:

Is anyone aware of why a notebook would take 2 and a half minutes when the code itself executed in the notebook takes little more than 1s? Can this be improved on by changing some configuration setting(s)?

Re: Performance and execution times

mikeburek — Tue, 20 Aug 2024 14:41:41 GMT

I'm just guessing here - If you rerun it right away, does it still take the same amount of time? If it has been awhile since you last ran code, it could be waiting for an instance to spin up?

Re: Performance and execution times

PedroCSeq — Tue, 20 Aug 2024 14:48:42 GMT

Times are not consistent, but the fact that it would take a lot less time to just run the code

But the efficiency and idle times in a one operation notebook shouldn't be this low and high respectivelly

Re: Performance and execution times

mikeburek — Tue, 20 Aug 2024 15:03:43 GMT

Oh, that is weird that your total idle time is still almost 80% of the total time. I didn't expect that to stay that high.

Re: Performance and execution times

PedroCSeq — Tue, 20 Aug 2024 15:08:55 GMT

The idle time is high basically across pipeline execution, this in a completely different notebook, so efficiency is not a given, but the iddle time is still high

Re: Performance and execution times

Anonymous — Wed, 21 Aug 2024 01:31:26 GMT

Hi PedroCSeq

It seems that you are experiencing a significant delay due to free time while running your laptop in Microsoft Fabric. This can often be attributed to the time it takes to start and manage a Spark session. You can try the following:

Enable high concurrency mode. This mode allows multiple notebooks to share a single Spark session, reducing the overhead of starting a new session each time.

With High Concurrency Mode enabled, there's no need to start new spark sessions every time to run a notebook.

Configure high concurrency mode for notebooks - Microsoft Fabric | Microsoft Learn

Use Spark Autotune. Fabric's Spark Autotune feature automatically fine-tunes Spark Settings to optimize performance and reduce execution time without human intervention.

Fabric Spark Autotune and Run Series Job Analysis | Microsoft Fabric Blog | Microsoft Fabric

Utilize the built-in Spark advisor in Fabric to analyze your notebook code and Spark executions in real-time.

Notebook contextual monitoring and debugging - Microsoft Fabric | Microsoft Learn

Ensure that your Spark configurations are optimized for the workload.

By implementing these strategies, you should be able to reduce the idle time and improve the overall efficiency of your notebook executions.

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.