topic Re: mssparkutils inside python package in Data Engineering

mssparkutils inside python package

bcdobbs — Thu, 30 Nov 2023 23:06:54 GMT

Hi,

I've started to build a set of helper functions for my fabric notebooks to do things like refreshes and getting attachments from emails. Mostly just wrappers round msal and requests.

Code runs fine direct in a notebook. I've then wrapped it in a wheel file and imported which again works. However when executing the imported code, the notebook says mssparkutils isn't recognised. Any ideas hiw I reference it correctly?

Code is in:

https://github.com/ArchwayTrust/Fabric-PythonHelper

thanks

Ben

Re: mssparkutils inside python package

Anonymous — Fri, 01 Dec 2023 05:56:16 GMT

Hi @bcdobbs
Thanks for using Fabric Community.
Have you run the code in Fabric Notebook? Did you install some other whl which includes notebookutils package?
How did you execute the imported code? Can you please provide me some screenshots along with the error message?
This would help me in understanding the issue better.
Thanks.

Re: mssparkutils inside python package

bcdobbs — Fri, 01 Dec 2023 10:32:51 GMT

I've gone back and built a very simple example.

Inside a notebook I can run:

def run_me_local(): help(mssparkutils) run_me_local()

and it happily returns the help description for mssparkutils.

If I wrap the same thing in a python wheel package and import it into an environment:

My question is how do I build a wheel file that "knows" about mssparkutils? In my local environment mssparkutils doesn't exist which I suspect is problem.

Full code used to produce the wheel:

my_code.py:

def run_me(): help(mssparkutils)

setup.py (guessing I need mssparkutils in the install_requires but it then says it doesn't exist:

from setuptools import setup, find_packages setup( name='wheel_test', version='0.1', packages=find_packages(), install_requires=[ ], author='Ben Dobbs', author_email='email@email.com', description='Wheel Testing', long_description=open('README.md').read(), long_description_content_type='text/markdown', )

Re: mssparkutils inside python package

Anonymous — Fri, 01 Dec 2023 11:07:16 GMT

Hi @bcdobbs
Thanks for explaining the issue.
Please install dummy notebookutils in local environment. This will solve your issue.

pip install dummy-notebookutils

For more information please refer to this link: dummy-notebookutils
Hope this helps. Please let me know if you have any further questions.

Re: mssparkutils inside python package

bcdobbs — Fri, 01 Dec 2023 11:15:58 GMT

Thanks I literally just found it works ok if I add

from notebookutils import mssparkutils

The dummy-notebookutils stops it complaining locally though so thank you for that.

Re: mssparkutils inside python package

Anonymous — Fri, 01 Dec 2023 11:20:59 GMT

Hi @bcdobbs
Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.