topic Re: Python Notebooks (not Pyspark) having default package issues in Data Engineering

Python Notebooks (not Pyspark) having default package issues

Dominoes2 — Mon, 09 Dec 2024 05:12:05 GMT

I have been working on a notebook today, went for lunch, came back and there are some pretty big changes that seem to have occured (Around 13:00 +10 😞

- Since then connecting to a session takes 60 seconds instead of 2 seconds in the morning

- Display() does not seem to work and fails silently

- Some packages that used to be in the environment now need to be installed with pip to be accessible such as deltalake

Is this all expected ?

Re: Python Notebooks (not Pyspark) having default package issues

Anonymous — Tue, 10 Dec 2024 01:16:27 GMT

HI @Dominoes2,

I'd like to suggest you check the default environment and spark pool settings if these settings changes. In addition, you could also check the fabric capacity SKU level if it reduced to a smaller level.

Create, configure, and use an environment in Fabric - Microsoft Fabric | Microsoft Learn

What is Power BI Premium? - Power BI | Microsoft Learn

Regards,

Xiaoxin Sheng

Re: Python Notebooks (not Pyspark) having default package issues

Dominoes2 — Tue, 10 Dec 2024 01:24:26 GMT

Hi @Anonymous

From what I can tell vanilla python notebooks do not support environments, only pyspark notebooks.

Use Python experience on Notebook - Microsoft Fabric | Microsoft Learn

The docs do say that for packages to work you may need to restart the kernal. This works when running it manually but how is this expected to work when running it on schedule? is there a programatic way to restart the kernal?

Re: Python Notebooks (not Pyspark) having default package issues

Tom_CA — Thu, 26 Dec 2024 22:27:13 GMT

Hello, guys; thank you for opening this conversation. I am facing the same situation here. I want to create a Python environment with some libraries. There is a simple code that I want to run, and I don't need Spark, just a Python environment. Do you know if we have another way to create an environment with some libraries that we need and are not in the available Python environment?

Re: Python Notebooks (not Pyspark) having default package issues

Anonymous — Mon, 30 Dec 2024 06:18:09 GMT

Hi @Dominoes2,

Perhaps you can tried to import the OS library in the notebooks and use its methods to operate with the kernel:

jupyter - Restart ipython Kernel with a command from a cell - Stack Overflow

Regards,

Xiaoxin Sheng