Forum Discussion
Performance and execution times
Hello everyone,
I'm building a lakehouse to serve as the centralized source of truth for an organization and the periodic ingestions are taking a looong time to ingest a relatively small volume of data. The built in fabric activities don't seem to be taking more that they should and the variance on the execution time is low, but the execution of notebooks in the pipelines seems to be broken:
Is anyone aware of why a notebook would take 2 and a half minutes when the code itself executed in the notebook takes little more than 1s? Can this be improved on by changing some configuration setting(s)?
Hi PedroCSeq
It seems that you are experiencing a significant delay due to free time while running your laptop in Microsoft Fabric. This can often be attributed to the time it takes to start and manage a Spark session. You can try the following:
Enable high concurrency mode. This mode allows multiple notebooks to share a single Spark session, reducing the overhead of starting a new session each time.
With High Concurrency Mode enabled, there's no need to start new spark sessions every time to run a notebook.
Configure high concurrency mode for notebooks - Microsoft Fabric | Microsoft Learn
Use Spark Autotune. Fabric's Spark Autotune feature automatically fine-tunes Spark Settings to optimize performance and reduce execution time without human intervention.
Fabric Spark Autotune and Run Series Job Analysis | Microsoft Fabric Blog | Microsoft Fabric
Utilize the built-in Spark advisor in Fabric to analyze your notebook code and Spark executions in real-time.
Notebook contextual monitoring and debugging - Microsoft Fabric | Microsoft Learn
Ensure that your Spark configurations are optimized for the workload.
By implementing these strategies, you should be able to reduce the idle time and improve the overall efficiency of your notebook executions.
Regards,
Nono Chen
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
5 Replies
Hi PedroCSeq
It seems that you are experiencing a significant delay due to free time while running your laptop in Microsoft Fabric. This can often be attributed to the time it takes to start and manage a Spark session. You can try the following:
Enable high concurrency mode. This mode allows multiple notebooks to share a single Spark session, reducing the overhead of starting a new session each time.
With High Concurrency Mode enabled, there's no need to start new spark sessions every time to run a notebook.
Configure high concurrency mode for notebooks - Microsoft Fabric | Microsoft Learn
Use Spark Autotune. Fabric's Spark Autotune feature automatically fine-tunes Spark Settings to optimize performance and reduce execution time without human intervention.
Fabric Spark Autotune and Run Series Job Analysis | Microsoft Fabric Blog | Microsoft Fabric
Utilize the built-in Spark advisor in Fabric to analyze your notebook code and Spark executions in real-time.
Notebook contextual monitoring and debugging - Microsoft Fabric | Microsoft Learn
Ensure that your Spark configurations are optimized for the workload.
By implementing these strategies, you should be able to reduce the idle time and improve the overall efficiency of your notebook executions.
Regards,
Nono Chen
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
I'm just guessing here - If you rerun it right away, does it still take the same amount of time? If it has been awhile since you last ran code, it could be waiting for an instance to spin up?
Times are not consistent, but the fact that it would take a lot less time to just run the code
But the efficiency and idle times in a one operation notebook shouldn't be this low and high respectivelly
Oh, that is weird that your total idle time is still almost 80% of the total time. I didn't expect that to stay that high.
