Forum Discussion

PedroCSeq's avatar
PedroCSeq
Frequent Visitor
2 years ago
Solved

Performance and execution times

Hello everyone, I'm building a lakehouse to serve as the centralized source of truth for an organization and the periodic ingestions are taking a looong time to ingest a relatively small volume of...
Need Help
pipelines
Tips and Tricks
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi PedroCSeq 

     

    It seems that you are experiencing a significant delay due to free time while running your laptop in Microsoft Fabric. This can often be attributed to the time it takes to start and manage a Spark session. You can try the following:

     

    Enable high concurrency mode. This mode allows multiple notebooks to share a single Spark session, reducing the overhead of starting a new session each time.

     

    With High Concurrency Mode enabled, there's no need to start new spark sessions every time to run a notebook.

     

    Configure high concurrency mode for notebooks - Microsoft Fabric | Microsoft Learn

     

    Use Spark Autotune. Fabric's Spark Autotune feature automatically fine-tunes Spark Settings to optimize performance and reduce execution time without human intervention.

     

    Fabric Spark Autotune and Run Series Job Analysis | Microsoft Fabric Blog | Microsoft Fabric

     

    Utilize the built-in Spark advisor in Fabric to analyze your notebook code and Spark executions in real-time.

     

    Notebook contextual monitoring and debugging - Microsoft Fabric | Microsoft Learn

     

    Ensure that your Spark configurations are optimized for the workload.

     

    By implementing these strategies, you should be able to reduce the idle time and improve the overall efficiency of your notebook executions. 

     

    Regards,

    Nono Chen

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

mikeburek's avatar
mikeburek
Advocate III
2 years ago

I'm just guessing here - If you rerun it right away, does it still take the same amount of time? If it has been awhile since you last ran code, it could be waiting for an instance to spin up?

PedroCSeq's avatar
PedroCSeq
Frequent Visitor
2 years ago

Times are not consistent, but the fact that it would take a lot less time to just run the code

But the efficiency and idle times in a one operation notebook shouldn't be this low and high respectivelly

  • mikeburek's avatar
    mikeburek
    Advocate III
    2 years ago

    Oh, that is weird that your total idle time is still almost 80% of the total time. I didn't expect that to stay that high.

    • PedroCSeq's avatar
      PedroCSeq
      Frequent Visitor
      2 years ago

      The idle time is high basically across pipeline execution, this in a completely different notebook, so efficiency is not a given, but the iddle time is still high