Performance and execution times

Hi PedroCSeq 

It seems that you are experiencing a significant delay due to free time while running your laptop in Microsoft Fabric. This can often be attributed to the time it takes to start and manage a Spark session. You can try the following:

Enable high concurrency mode. This mode allows multiple notebooks to share a single Spark session, reducing the overhead of starting a new session each time.

With High Concurrency Mode enabled, there's no need to start new spark sessions every time to run a notebook.

Configure high concurrency mode for notebooks - Microsoft Fabric | Microsoft Learn

Use Spark Autotune. Fabric's Spark Autotune feature automatically fine-tunes Spark Settings to optimize performance and reduce execution time without human intervention.

Fabric Spark Autotune and Run Series Job Analysis | Microsoft Fabric Blog | Microsoft Fabric

Utilize the built-in Spark advisor in Fabric to analyze your notebook code and Spark executions in real-time.

Notebook contextual monitoring and debugging - Microsoft Fabric | Microsoft Learn

Ensure that your Spark configurations are optimized for the workload.

By implementing these strategies, you should be able to reduce the idle time and improve the overall efficiency of your notebook executions. 

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.