Forum Discussion

smpa01's avatar
smpa01
Community Champion
1 year ago
Solved

Seeking advice on custom packages

TLDR: if a custom python package has an executable, how can I have the package worked in notebook the same way it works locally

 

I have ben developing notebooks by creating custom environments and custom packages. Some of my packages were built by using cython. So the whl content looks like this

for cython 

custom_cython_package
 ┣ basic_cy.cp313-win_amd64.pyd
 ┗ __init__.py

vs non-cython

ZZ_TKN
 ┣ scripts.py
 ┗ __init__.py

 

The cython package works as desired once intalled and tested locally

 

But the same package fails in databricks notebook

 

Can anyone please explain why it fails, I am not sure if itis due to C compiler / MS C++ Buid tols which I have in my local environment but fabric environment lacks it and .pyd requires an interpreter there?

 

Is there a workaround for cythonize py packages to work in notebook?

 

Thank you in advance.

Environment
Notebook

4 Replies

  • v-nmadadi-msft's avatar
    v-nmadadi-msft
    Community Support
    1 year ago

    Hi smpa01,
    Thanks for reaching out to the Microsoft fabric community forum.

    In the image you shared, you are using "env" in the path of the .whl file. Instead, please use "test_env" to ensure the correct environment is referenced. 

    Even after trying the above mentioned point if it still fails re try the pip installation process using mssparkutils.nbResPath

    mssparkutils.nbResPath is a function within the Microsoft Spark Utilities library, designed to assist in accessing resources uploaded to a notebook in Microsoft Fabric. It provides the path to these resources, facilitating their utilization within the notebook environment.

    path = mssparkutils.nbResPath + "/test_env/package_name.whl"

!pip install path


    If you find this post helpful, please mark it as an "Accept as Solution" and consider giving a KUDOS.
    Thanks and Regards

    • smpa01's avatar
      smpa01
      Community Champion
      1 year ago

      1 thing to point out here the name of the environment probably has nothing to do with pip install

       

       

       

      This is the content of

      ```

      app-0.1.0-cp313-cp313-win_amd64.whl

      ```

      +---app
|       core.cp313-win_amd64.pyd
|       main.py
|       __init__.cp313-win_amd64.pyd
|
\---app-0.1.0.dist-info
        METADATA
        RECORD
        top_level.txt
        WHEEL

      I am not sure  if whl with pyd works

  • v-nmadadi-msft's avatar
    v-nmadadi-msft
    Community Support
    1 year ago

    Hi smpa01 ,

    Could you please confirm if you've submitted this as an idea in the Ideas Forum? If so, sharing the link here would be helpful for other community members who may have similar feedback.

    If we don’t hear back, we’ll go ahead and close this thread. For any further discussions or questions, please start a new thread in the Microsoft Fabric Community Forum  we’ll be happy to assist.

    Thank you for being part of the Microsoft Fabric Community.