Intellisense for custom Python packages not working in Fabric notebooks

[HoneHealthMB]

I have built a very simple Python package to use in my Fabric notebooks.  The package is basically just a set of some utility/helper classes.  I have searched extensively, and attempted countless different configurations to get intellisense to work within the Fabric notebooks - but everything I have tried has failed.

Goal:  Get my custom package's classes and methods to show their docstrings as expected.

I have tried all various types of "official" commenting methods/syntaxes:

```

def parse_datetime_string(self, date_str:str):

        """

        Safely parse an datetime string to a datetime object,

        properly handling timezone information.

        Args:

            date_str (str): The date string to parse (various formats supported)

        Returns:

            tuple: (datetime_obj, iso_format_str) - Both could be None if parsing fails

        """

```

That didn't work - so I went and looked at Microsoft's own packages (azure-ai-projects for example) - and they follow a different format, so I tried that:

    def get_doc_contents(self, doc_id: str, show_progress: bool = True😞

        """

        Download a Google Doc as DOCX using a service account with domain-wide delegation.

        Note: I initially attempted to have a generic `get_file_contents` method

        :param doc_id: The ID of the Google Doc (found in the URL)

        :type doc_id: str

        :param show_progress: Whether or not to show progress as it downloads the file

        :type show_progress: bool

        :return: BytesIO object containing the DOCX file content

        :rtype: tuple[io.BytesIO, str]

        """    

Then I realized that even the azure packages themselves do not render good intellisense inside Fabric notebooks.

I've tried all the normal tricks for how wheel packages are built - no dice.

• If I add my custom wheel package to my custom Spark environment, it doesn't work.
• If I add the wheel package into the notebook's resources files, and then do a %pip install - it doesn't work
• If I just put the code itself directly into the notebook, it does work - but that's useless and defeats the purpose

To be clear, I *am* able to see the definition of the method and the names of the parameters - but I do not see any of the detailed description of the method or the parameters or the classes (i.e. the stuff that is actually helpful to people).

I did make one interesting discovery - and that is the bs4 package (BeautifulSoup) actually renders the rich intellisense content amazingly well.

Has anyone managed to do this successfully?

[v-ssriganesh]

Hi @HoneHealthMB,
Thank you for reaching out to the Microsoft fabric community forum.

After thoroughly reviewing the details you provided, it seems the core issue is that Pylance (the IntelliSense engine in Fabric notebooks) isn’t rendering the detailed docstrings for your custom package when installed as a wheel, either via a custom Spark environment or notebook resources. The fact that IntelliSense works when the code is directly in the notebook, and that bs4 displays rich IntelliSense, suggests this may be related to how Pylance processes packaged libraries in Fabric’s Spark environment.

Here are a few suggestions to try:

• Ensure your package includes type hints and possibly type stub files. Pylance relies heavily on these for rich IntelliSense. The success of bs4 likely stems from its well-structured metadata, which you could try to emulate in your package structure.
• As a workaround, you could upload your package’s source files to the notebook’s resources folder and add the folder to the Python path. This mimic placing code directly in the notebook and may allow Pylance to display docstrings, though I understand this isn’t ideal for your use case.
• When installing your wheel package (via %pip install or a custom environment), verify that it’s correctly installed in the session’s library path. Restarting the Spark session after installation might help ensure the package is properly loaded.
• Since bs4 works well, analyzing its package structure could provide insights. This might involve looking at its installed files to identify what makes its IntelliSense rendering effective.

For additional guidance on managing Python packages in Fabric notebooks, you may find this official documentation helpful: Using Python Experience on Notebook. It covers package installation and environment setup, which might provide further context.

The fact that even Microsoft’s azure-ai-projects package doesn’t render rich IntelliSense suggests this could be a limitation in Fabric’s Pylance integration, possibly due to its preview status.

If this information is helpful, please “Accept as solution” and give a "kudos" to assist other community members in resolving similar issues more efficiently.
Thank you.

[HoneHealthMB]

Can we get one of the fabric devs to chime in on this?

The reality is that many libraries out there have intellisense working just fine.  So it isn't a limitation of the tool.  Those libraries are all also installed via packages.  pandas is an example.  bs4 is another example.

The question is very simple:  what is the Fabric UI code trying to do when it loads intellisense/docstrings?  You can clearly see it doing a little "loading..." thing for a split second - so where is it trying to get that info from?

[v-ssriganesh]

Hi @HoneHealthMB,

Thank you for your follow-up and for pointing out that libraries like pandas and bs4 render IntelliSense well, which shows this isn’t a limitation of Fabric’s tools. Your observation of the “Loading…” message in the UI is a great clue it suggests Pylance is trying to fetch docstring data but not succeeding for your custom wheel package.

The “Loading…” likely indicates Pylance is querying your package’s metadata or files in the Spark environment’s library path, possibly missing type information (e.g: type stubs or a py.typed file) that pandas and bs4 include. Here are some next steps to try:

• Check if pandas or bs4 include type-related files your package might need to enable rich IntelliSense.
• Ensure your custom Spark environment is configured correctly, as outlined in the Using Python Experience on Notebook page. Restarting the session might help Pylance.
• Temporarily using source files in the notebook’s resources folder could confirm if Pylance reads docstrings differently from your wheel package.

If this response helps, consider marking it as “Accept as solution” and giving a “kudos” to assist other community members.
Thank you.

[HoneHealthMB]

I'm fairly new to python - is there a relatively straightforward way to determine precisely what is in the bs4 package?

[v-ssriganesh]

Hello @HoneHealthMB,

Even if you're newer to Python, there are a few straightforward ways to explore what’s included in the bs4 package, even if you’re new to Python.

One easy approach is to locate where the package is installed in your environment and then browse its contents. You can do this either from within the notebook using basic Python tools (like checking the file path of an imported package) or by exploring the installed packages folder directly via the notebook file browser or environment settings.

You're specifically looking for:

• A py.typed file — which marks the package as having type information
• Any .pyi stub files — these provide explicit type definitions that tools like IntelliSense can use
• Well-documented source files with structured docstrings and type hints
  
  

Another helpful option is to review the package on GitHub or PyPI. For example, bs4 is available on GitHub, where you can see how its files and docstrings are organized.

If this reply helps, please consider marking it as "Accept as solution" and giving it a kudos to support others in the community.

[HoneHealthMB]

Thanks.  I have actually done all of those things exhaustively.  I have a py.typed file in my package, all stub filres are in place.  Docstrings are in place and type hints are there.

The intellisense *will* show the type hints - the issue is that it doesn't load any of the commentary around how the methods work, or what they do, etc, etc.

I have even downloaded the bs4 wheel package, given it to chat GPT, adn then given it my wheel package.  I'm literally doing everything exactly the same way.  The only exception is that I was including .stub files and they do not - so I am going to try and remove them.

[v-ssriganesh]

Hi @HoneHealthMB,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

[HoneHealthMB]

I have an update that I hope helps.

I decided to simplify this as much as possible and found that the error is almost definitely a bug in the Fabric notebook UI.

After extensive comparisons between my library and others - it is clear that there is no ryhme or reason as to what makes it work or not.  I created a blank notebook in a default environment, with no customizations.

I did a pip install of my module.  No docstrings are rendered, and half of the time it isn't even correctly identifying what is a "class" and what is an "Any" type - even at the module level.  It has this same problem with the azure libraries.

I then went to the Google Colab notebook tools, and repeated the same process.  All docstrings are rendered perfectly, without issue, without any extra steps.  That is even true of the azure libraries themselves.

I think at this point, if Google Colab is better at recognizing Microsoft's intellisense than its own editor - we can say with a fair amount of certainty that there is a legitimate bug somewhere.  My guess is that it is in how it is caching and what not - but even if I eradicate my local cache - it makes no difference.

Both Google Colab and the Fabric notebooks use the monaco code editor FYI.

Do we have enough info at this point to escalate to someone at MS?

[v-ssriganesh · Accepted solution]

Hi @HoneHealthMB,
We apologize for the inconvenience caused. Please consider raising a Microsoft support ticket for further investigation. You can explain all the troubleshooting steps you have taken to help them better understand the issue.

You can create a Microsoft support ticket with the help of the link below:
https://learn.microsoft.com/en-us/power-bi/support/create-support-ticket

If this helps, please "Accept as solution" and drop a "Kudos" so other members can find it more easily.

Thank you.

[v-ssriganesh]

Hi @HoneHealthMB,
Could you please confirm if the issue has been resolved after raising a support case? If a solution has been found, it would be greatly appreciated if you could share your insights with the community. This would be helpful for other members who may encounter similar issues.

Thank you for your understanding and assistance.

[HoneHealthMB]

I have tried using stub files and they make no difference.  I have tried every format of docstrings.  I compared every potential difference between the bs4 wheel package and mine.  There are no differences.

At this point, I am hoping that someone at Microsoft who works on the Fabric UI can speak to this.  

[HoneHealthMB]

Sorry, hit reply too soon...

I have dug deeply into the bs4 code and azure-ai-projects code and I cannot find any differences in how the docstrings are formatted.  Since its a wheel package, I do not know any way to force it to include additional help files.  I did apply the "zip_safe=False" flag as I thought that might be the issue.

I have discovered that the Fabric notebook UI is at least attempting to load the help/docstring content - because when you hover over a class name or method name, you will see a "Loading.." message appear very briefly right under the popover method name.  The trick now is to try and figure out where it is attempting to load it from.

[HoneHealthMB]

Thank you for the reply.

I am adding my wheel package to a custom Spark environment in Fabric to avoid our engineers from having to install it on every notebook, etc.    If I do this in my notebook:

print(myclass.__doc__)

I see the documentation content just fine.