topic Re: Starting Session takes really slow - notebook session Microsoft Fabric in Data Engineering

Starting Session takes really slow - notebook session Microsoft Fabric

innocence84 — Mon, 12 May 2025 19:53:00 GMT

Hi all,

We are using F32 Fabric Capacity. When we use streaming environment, calling a notebook from a pipeline takes long time. When I open the notebook and click start session, it also takes a long time.

When I open a notebook and select "Workspace Default" it is really fast.

However when I call the notebook from a pipeline, I took this error message below:

Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - MagicUsageError, Error value - %pip magic command is disabled. Find more details in https://learn.microsoft.com/en-us/fabric/data-engineering/library-management#inline-installation' :

How can I call this notebook, that uses Workspace default, from a pipeline?

Re: Starting Session takes really slow - notebook session Microsoft Fabric

nilendraFabric — Mon, 12 May 2025 22:31:16 GMT

Hi @innocence84

The error occurs because inline package installation (`%pip`) is disabled in non-interactive pipeline runs.

Use Fabric Environments to pre-install libraries:
1. Create a Fabric Environment in your workspace.
2. Add required Python libraries (e.g., `semantic-link`, `geopandas`) to the environment.
3. Publish the environment and set it as the default for your workspace or pipeline

Optimize session startup using High Concurrency Mode and default environments.

Re: Starting Session takes really slow - notebook session Microsoft Fabric

v-saisrao-msft — Tue, 13 May 2025 08:54:37 GMT

Hi @innocence84,

Thank you for reaching out to the Microsoft Fabric Forum Community.

The response which provided by @nilendraFabric, identifies that the %pip is disabled in non-interactive pipeline runs. Try using Fabric Environments to pre-install the required libraries and assign the environment to your notebook pipeline or set it as the default workspace environment.

Additionally, also provided the Microsoft links for your reference:

Configure high concurrency mode for notebooks - Microsoft Fabric | Microsoft Learn.

Known issue - Slow Spark session startup caused by Native Execution Engine - Microsoft Fabric | Microsoft Learn.

If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

Thank you.

Re: Starting Session takes really slow - notebook session Microsoft Fabric

v-saisrao-msft — Fri, 16 May 2025 05:19:16 GMT

Hi @innocence84,
I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.
Thank you.

Re: Starting Session takes really slow - notebook session Microsoft Fabric

innocence84 — Fri, 16 May 2025 14:05:28 GMT

So is high concurrency session is only solution to the slow startup issue? And what is the pricing of the high concurrency session

Re: Starting Session takes really slow - notebook session Microsoft Fabric

v-saisrao-msft — Mon, 19 May 2025 08:50:21 GMT

Hi @innocence84,

Hello there
There's no extra fee for High Concurrency Mode it comes with your F32 Fabric Capacity. It consumes your current capacity units (CU), so you can turn it on and utilize it without paying extra.

If this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.

Thank you.