Forum Discussion

HoneHealthMB's avatar
HoneHealthMB
Icon for Advocate I rankAdvocate I
1 year ago
Solved

Intellisense for custom Python packages not working in Fabric notebooks

I have built a very simple Python package to use in my Fabric notebooks.  The package is basically just a set of some utility/helper classes.  I have searched extensively, and attempted countless different configurations to get intellisense to work within the Fabric notebooks - but everything I have tried has failed.

 

Goal:  Get my custom package's classes and methods to show their docstrings as expected.

 

I have tried all various types of "official" commenting methods/syntaxes:

   
```
def parse_datetime_string(self, date_str:str):
        """
        Safely parse an datetime string to a datetime object,
        properly handling timezone information.
       
        Args:
            date_str (str): The date string to parse (various formats supported)
           
        Returns:
            tuple: (datetime_obj, iso_format_str) - Both could be None if parsing fails
        """

```

That didn't work - so I went and looked at Microsoft's own packages (azure-ai-projects for example) - and they follow a different format, so I tried that:

 

    def get_doc_contents(self, doc_id: str, show_progress: bool = TruešŸ˜ž
        """
        Download a Google Doc as DOCX using a service account with domain-wide delegation.
        Note: I initially attempted to have a generic `get_file_contents` method

        :param doc_id: The ID of the Google Doc (found in the URL)
        :type doc_id: str
        :param show_progress: Whether or not to show progress as it downloads the file
        :type show_progress: bool
        :return: BytesIO object containing the DOCX file content
        :rtype: tuple[io.BytesIO, str]
        """    

 

Then I realized that even the azure packages themselves do not render good intellisense inside Fabric notebooks.

 

I've tried all the normal tricks for how wheel packages are built - no dice.

 

  • If I add my custom wheel package to my custom Spark environment, it doesn't work.
  • If I add the wheel package into the notebook's resources files, and then do a %pip install - it doesn't work
  • If I just put the code itself directly into the notebook, it does work - but that's useless and defeats the purpose

To be clear, I *am* able to see the definition of the method and the names of the parameters - but I do not see any of the detailed description of the method or the parameters or the classes (i.e. the stuff that is actually helpful to people).

 

I did make one interesting discovery - and that is the bs4 package (BeautifulSoup) actually renders the rich intellisense content amazingly well.

 

Has anyone managed to do this successfully?

 

 

 

 

data engineering
environment
  • v-ssriganesh's avatar
    v-ssriganesh
    1 year ago

    Hi HoneHealthMB,
    We apologize for the inconvenience caused. Please consider raising a Microsoft support ticket for further investigation. You can explain all the troubleshooting steps you have taken to help them better understand the issue.

    You can create a Microsoft support ticket with the help of the link below:
    https://learn.microsoft.com/en-us/power-bi/support/create-support-ticket

    If this helps, please "Accept as solution" and drop a "Kudos" so other members can find it more easily.

    Thank you.

     

13 Replies

  • v-ssriganesh's avatar
    v-ssriganesh
    Icon for Community Support rankCommunity Support
    1 year ago

    Hi HoneHealthMB,
    Thank you for reaching out to the Microsoft fabric community forum.

    After thoroughly reviewing the details you provided, it seems the core issue is that Pylance (the IntelliSense engine in Fabric notebooks) isnā€™t rendering the detailed docstrings for your custom package when installed as a wheel, either via a custom Spark environment or notebook resources. The fact that IntelliSense works when the code is directly in the notebook, and that bs4 displays rich IntelliSense, suggests this may be related to how Pylance processes packaged libraries in Fabricā€™s Spark environment.

    Here are a few suggestions to try:

    • Ensure your package includes type hints and possibly type stub files. Pylance relies heavily on these for rich IntelliSense. The success of bs4 likely stems from its well-structured metadata, which you could try to emulate in your package structure.
    • As a workaround, you could upload your packageā€™s source files to the notebookā€™s resources folder and add the folder to the Python path. This mimic placing code directly in the notebook and may allow Pylance to display docstrings, though I understand this isnā€™t ideal for your use case.
    • When installing your wheel package (via %pip install or a custom environment), verify that itā€™s correctly installed in the sessionā€™s library path. Restarting the Spark session after installation might help ensure the package is properly loaded.
    • Since bs4 works well, analyzing its package structure could provide insights. This might involve looking at its installed files to identify what makes its IntelliSense rendering effective.

    For additional guidance on managing Python packages in Fabric notebooks, you may find this official documentation helpful: Using Python Experience on Notebook. It covers package installation and environment setup, which might provide further context.

    The fact that even Microsoftā€™s azure-ai-projects package doesnā€™t render rich IntelliSense suggests this could be a limitation in Fabricā€™s Pylance integration, possibly due to its preview status.


    If this information is helpful, please ā€œAccept as solutionā€ and give a "kudos" to assist other community members in resolving similar issues more efficiently.
    Thank you.

    • HoneHealthMB's avatar
      HoneHealthMB
      Icon for Advocate I rankAdvocate I
      1 year ago

      Thank you for the reply.

       

      I am adding my wheel package to a custom Spark environment in Fabric to avoid our engineers from having to install it on every notebook, etc.    If I do this in my notebook:

       

      print(myclass.__doc__)

       

      I see the documentation content just fine. 

    • HoneHealthMB's avatar
      HoneHealthMB
      Icon for Advocate I rankAdvocate I
      1 year ago

      Sorry, hit reply too soon...

       

      I have dug deeply into the bs4 code and azure-ai-projects code and I cannot find any differences in how the docstrings are formatted.  Since its a wheel package, I do not know any way to force it to include additional help files.  I did apply the "zip_safe=False" flag as I thought that might be the issue.

       

      I have discovered that the Fabric notebook UI is at least attempting to load the help/docstring content - because when you hover over a class name or method name, you will see a "Loading.." message appear very briefly right under the popover method name.  The trick now is to try and figure out where it is attempting to load it from.

    • HoneHealthMB's avatar
      HoneHealthMB
      Icon for Advocate I rankAdvocate I
      1 year ago

      Can we get one of the fabric devs to chime in on this?

       

      The reality is that many libraries out there have intellisense working just fine.  So it isn't a limitation of the tool.  Those libraries are all also installed via packages.  pandas is an example.  bs4 is another example.

       

      The question is very simple:  what is the Fabric UI code trying to do when it loads intellisense/docstrings?  You can clearly see it doing a little "loading..." thing for a split second - so where is it trying to get that info from?

      • v-ssriganesh's avatar
        v-ssriganesh
        Icon for Community Support rankCommunity Support
        1 year ago

        Hi HoneHealthMB,

        Thank you for your follow-up and for pointing out that libraries like pandas and bs4 render IntelliSense well, which shows this isnā€™t a limitation of Fabricā€™s tools. Your observation of the ā€œLoadingā€¦ā€ message in the UI is a great clue it suggests Pylance is trying to fetch docstring data but not succeeding for your custom wheel package.

        The ā€œLoadingā€¦ā€ likely indicates Pylance is querying your packageā€™s metadata or files in the Spark environmentā€™s library path, possibly missing type information (e.g: type stubs or a py.typed file) that pandas and bs4 include. Here are some next steps to try:

        • Check if pandas or bs4 include type-related files your package might need to enable rich IntelliSense.
        • Ensure your custom Spark environment is configured correctly, as outlined in the Using Python Experience on Notebook page. Restarting the session might help Pylance.
        • Temporarily using source files in the notebookā€™s resources folder could confirm if Pylance reads docstrings differently from your wheel package.


        If this response helps, consider marking it as ā€œAccept as solutionā€ and giving a ā€œkudosā€ to assist other community members.
        Thank you.