Forum Discussion

P_work's avatar
P_work
Helper I
7 months ago

Python 3.11 notebook slow startup when using builtin import

Python 3.11 notebook is slow to startup when using builtin import.  If builtin py file imports are not used, the notebook is orders of magnitude faster startup. Are there no workarounds to handle this? Most of these type of notebooks will be called in pipelines, so not sure if possible to 'warm' or always cold start?

data engineering

6 Replies

  • deborshi_nag's avatar
    deborshi_nag
    Super User
    7 months ago

    Hello P_work 

     

    Use Fabric Environments item to install and govern Python libraries (including your own custom modules) for notebooks. This process gives you repeatability, version control, and attach-at-once behavior across notebooks/Spark job definitions. Keep %pip inline installs for ad‑hoc or exploratory work, not for anything you want to be reliable or reusable.
     
    Use the following rules for custom libraries -
     
    1. Team‑wide, repeatable libraries for notebooks/Jobs
    Create a Fabric Environment, add libraries (PyPI/conda, private feed, or custom wheels), and attach it (or set as Workspace default).
     
    2. Quick experiments in a single notebook session
    Use %pip install ... in a cell.
     
    3. Your own modules (internal utilities)
    Package as a Wheel (.whl) and add it as a custom library in the Environment.
     
    Please refer to the following Microsoft doc reference
     
    Hope this helps - please appreciate leaving a Kudos or accepting as a Solution!
    • P_work's avatar
      P_work
      Helper I
      7 months ago

      I have implented custom environments add found the startup times to be excessively long as well. This is a known issue from what I have seen in the community and will not alleviate long session startups. When anything other than the default environment is used, especially from pipelines, startup times will be orders of magnitude longer.

  • v-hjannapu's avatar
    v-hjannapu
    Community Support
    7 months ago

    Hi P_work,

    Thank you  for reaching out to the Microsoft fabric community forum.

    When a notebook is triggered from a pipeline, it always starts with a fresh session. During startup, Python has to set up the environment and load all imported files. If the notebook is using built-in or shared .py imports, this step takes more time, so the notebook startup feels slow. When those imports are removed, startup becomes faster.

    Currently, there is no option to keep the notebook warm or reuse the same session when running from pipelines. Cold start will happen every time.

    As a workaround, try to keep only required imports at the top and move other imports inside functions. Also, keep shared utility files as small and simple as possible to reduce startup delay.

    Once the notebook starts, execution speed should be normal. The delay is mainly during initialization.

    Hope this clarifies the issue.
    Regards,
    Community Support Team.

    • v-hjannapu's avatar
      v-hjannapu
      Community Support
      7 months ago

      Hi P_work,
      I hope the information provided above assists you in resolving the issue. If you have any additional questions or concerns, please do not hesitate to contact us. We are here to support you and will be happy to help with any further assistance you may need.

      Regards,
      Community Support Team.