topic Re: Using custom python library in notebook in Data Engineering

Using custom python library in notebook

Broeks — Sun, 20 Oct 2024 10:25:02 GMT

I've create a whl file containing a python function and uploaded it as a custom library into my fabric environment. I've succesfully deployed the library and now I would expect the library to be available to import during a notebook session.

However, this is not the case. When I run the pip list, it's not shown in the list and I can also not call the library through the import function:

When I install the package from the whl file in the environment resource, the package does work. However I would expect to just add my whl file to the environment as a custom library and then be able to use the packages inside. Did I miss something or do i need the use the custom library in another way?

Re: Using custom python library in notebook

Anonymous — Mon, 21 Oct 2024 01:23:00 GMT

Hi @Broeks ,

Can you please share some more detail information about this issue and sameple whl file ? They should help us clarify your scenario and test to troubleshoot.

In addition, have you tried to use %pip list instead of native jupyter pip command to list the installed libraries?

Regards,

Xiaoxin Sheng

Re: Using custom python library in notebook

Srisakthi — Mon, 21 Oct 2024 11:42:03 GMT

Hi @Broeks ,

Are you able to see your library in workspace settings of Spark?

Thanks,

Srisakthi

Re: Using custom python library in notebook

Broeks — Mon, 21 Oct 2024 16:59:06 GMT

Thanks for your responses.
To answer @Srisakthi, yes the library is present:

And to @Anonymous, i've tried the %pip list and my package isn't present.

Some additional info i can give is:

- I've created the package my self, based on een small set of python files. However, since the package does work when i install it through a notebook cell, I''ve concluded that the package isn't the issue.

This is the structure and one of the functions is the package:

This is the code i've used for the creation of the package:

from setuptools import find_packages, setup setup( name='idpdemo', version='0.0.1', description='demo_description', author='Broeks', packages=find_packages(include=['idp']), install_requires=[], # Add your dependencies here extras_require={ 'dev': ['pytest'] # Test dependencies go here } )

And I've tried to import the function through the following code:

from idp import check_dataframe_for_duplicate_keys

If you need additional information, let me known.

Re: Using custom python library in notebook

Anonymous — Tue, 22 Oct 2024 06:06:23 GMT

HI @Broeks,

I'd like to suggest text with other whl format custom libraries to confirm if this issue only applies on your side.

Regards,

Xiaoxin Sheng

Re: Using custom python library in notebook

Broeks — Tue, 22 Oct 2024 07:42:11 GMT

Yes, will do. I'm quitte busy these days, so a response might take a few days. But thanks for the help so far!

Re: Using custom python library in notebook

Anonymous — Fri, 25 Oct 2024 08:34:14 GMT

HI @Broeks,

Sure, you can free feel to post here if you have any updates.

Regards,

Xiaoxin Sheng

Re: Using custom python library in notebook

jj44 — Thu, 12 Jun 2025 09:22:59 GMT

Hi @Broeks

Did you manage to solve this problem? I'm at the point in a project where writing a custom library maybe a better option, rather than executing notebook from within another notebook.

Cheers

Jeff Jones

Re: Using custom python library in notebook

Broeks — Thu, 12 Jun 2025 11:47:08 GMT

Hello Jeff,

Yes i was able to fix the issue. My package was not correct.

Now I'm able to upload and publish a whl file through the fabric api.

https://learn.microsoft.com/en-us/rest/api/fabric/environment/spark-libraries

Re: Using custom python library in notebook

jj44 — Thu, 12 Jun 2025 13:01:30 GMT

Thanks, good to hear.

I might put some time in and see if I can migrate my code to a library then.

Cheers,

Jeff Jones

Re: Using custom python library in notebook

Broeks — Thu, 12 Jun 2025 14:08:27 GMT

Yes, do you have any idea on how you are going to develop your python/pysparkcode?

I wanted to use the spark job definition integration within vscode to develop against the fabric kernel. However, debugging doesn't work as expected.

https://marketplace.visualstudio.com/items?itemName=SynapseVSCode.synapse#:~:text=The%20Fabric%20Data%20Engineering%20extension%20%28Previously%20Synapse%20VS,extension%20is%20available%20from%20the%20Visual%20Studio%20Marketplace.

So I'm wondering whats your approach?