Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified in Microsoft Fabric—for free! For a limited time, the Microsoft Fabric Community team will be offering free DP-600 exam vouchers. Prepare now

Reply
PedroCSeq
Frequent Visitor

Performance and execution times

Hello everyone,

I'm building a lakehouse to serve as the centralized source of truth for an organization and the periodic ingestions are taking a looong time to ingest a relatively small volume of data. The built in fabric activities don't seem to be taking more that they should and the variance on the execution time is low, but the execution of notebooks in the pipelines seems to be broken:

PedroCSeq_1-1724147367478.pngPedroCSeq_2-1724147382284.png

PedroCSeq_3-1724147523669.png

 

Is anyone aware of why a notebook would take 2 and a half minutes when the code itself executed in the notebook takes little more than 1s? Can this be improved on by changing some configuration setting(s)?

1 ACCEPTED SOLUTION
v-nuoc-msft
Community Support
Community Support

Hi @PedroCSeq 

 

It seems that you are experiencing a significant delay due to free time while running your laptop in Microsoft Fabric. This can often be attributed to the time it takes to start and manage a Spark session. You can try the following:

 

Enable high concurrency mode. This mode allows multiple notebooks to share a single Spark session, reducing the overhead of starting a new session each time.

 

With High Concurrency Mode enabled, there's no need to start new spark sessions every time to run a notebook.

 

Configure high concurrency mode for notebooks - Microsoft Fabric | Microsoft Learn

 

Use Spark Autotune. Fabric's Spark Autotune feature automatically fine-tunes Spark Settings to optimize performance and reduce execution time without human intervention.

 

Fabric Spark Autotune and Run Series Job Analysis | Microsoft Fabric Blog | Microsoft Fabric

 

Utilize the built-in Spark advisor in Fabric to analyze your notebook code and Spark executions in real-time.

 

Notebook contextual monitoring and debugging - Microsoft Fabric | Microsoft Learn

 

Ensure that your Spark configurations are optimized for the workload.

 

By implementing these strategies, you should be able to reduce the idle time and improve the overall efficiency of your notebook executions. 

 

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

View solution in original post

5 REPLIES 5
v-nuoc-msft
Community Support
Community Support

Hi @PedroCSeq 

 

It seems that you are experiencing a significant delay due to free time while running your laptop in Microsoft Fabric. This can often be attributed to the time it takes to start and manage a Spark session. You can try the following:

 

Enable high concurrency mode. This mode allows multiple notebooks to share a single Spark session, reducing the overhead of starting a new session each time.

 

With High Concurrency Mode enabled, there's no need to start new spark sessions every time to run a notebook.

 

Configure high concurrency mode for notebooks - Microsoft Fabric | Microsoft Learn

 

Use Spark Autotune. Fabric's Spark Autotune feature automatically fine-tunes Spark Settings to optimize performance and reduce execution time without human intervention.

 

Fabric Spark Autotune and Run Series Job Analysis | Microsoft Fabric Blog | Microsoft Fabric

 

Utilize the built-in Spark advisor in Fabric to analyze your notebook code and Spark executions in real-time.

 

Notebook contextual monitoring and debugging - Microsoft Fabric | Microsoft Learn

 

Ensure that your Spark configurations are optimized for the workload.

 

By implementing these strategies, you should be able to reduce the idle time and improve the overall efficiency of your notebook executions. 

 

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

mikeburek
Advocate II
Advocate II

I'm just guessing here - If you rerun it right away, does it still take the same amount of time? If it has been awhile since you last ran code, it could be waiting for an instance to spin up?

Times are not consistent, but the fact that it would take a lot less time to just run the code

PedroCSeq_0-1724165102573.png

PedroCSeq_1-1724165176123.png

But the efficiency and idle times in a one operation notebook shouldn't be this low and high respectivelly

Oh, that is weird that your total idle time is still almost 80% of the total time. I didn't expect that to stay that high.

The idle time is high basically across pipeline execution, this in a completely different notebook, so efficiency is not a given, but the iddle time is still high 

PedroCSeq_2-1724166493118.png

 

 

Helpful resources

Announcements
Oct Fabric Update Carousel

Fabric Monthly Update - October 2024

Check out the October 2024 Fabric update to learn about new features.

September Hackathon Carousel

Microsoft Fabric & AI Learning Hackathon

Learn from experts, get hands-on experience, and win awesome prizes.

October NL Carousel

Fabric Community Update - October 2024

Find out what's new and trending in the Fabric Community.