Notebooks, VS Code and Customized Python libraries

[SergioTorrinha]

Hi everyone!

I have a couple questions regarding notebooks, customized Python libraries and VS code (desktop app), for which I hope to seek your help.

1 - As far as I understood, the correct way of building customized Python libraries are through wheel files. However, it's quite cumbersome, when you need to do adjustments to the customized libraries to build wheel files everytime, not to mention the time to load those wheel files in a customized environment - the publishing times, from my experiences, can vary from 10 to 15 minutes each time.
Is there any better/more efficient/less cumbersome way to deal with customized libraries?
(And no, not having customized python libraries is not an option whatsoever 🙂 )

2 - Yesterday, while I was after an answer to question 1, I came accross a notebook functionality I didn't know: notebook resources, as mentioned in this documentation. The referenced documentation, for me it's not clear. Could perhaps someone point me out to a direction to better understand this functionality and how to use it, as it seems it can be an alternative to wheel files, please?

3 - I understand MS Fabric is in it's 'early begininggs' and, therefore, there are functionalities that are still evolving and this might be the case of notebooks, python environments and custom libraries. Is there a roadmap on what is planned, specifically, to this concepts?

It's hard to sell MS Fabric, internally,  when you come accross things that won't help you being productive, while you try to develop you data products and, for this reason, I apretiate in advance clarifications to these questions.
Also, please, let me know if, through reading my questions, you find something incorrect in the way I am looking/using notebooks, customized Python libraries and VS code (desktop app).

Thank you.

[SergioTorrinha]

Hi @v-nikhilan-msft !

Sorry for late response, but I was away for some Christmas time with family and friends. 🙂

Before anything, thanks for your input.
Based on it, I think that, for now, the most straightforward way of making use of custom libraries is really the notebook resource feature, because one only needs to publish the 'resource'/library into MS Fabric and that's it, which makes it easy and simple to use and to adjust whenever there is the need for it.

With this is mind, I have a couple more questions:

1. Is there any other documentation for it, excluding the one I pointed out initially? Is it planned to build some more documentation for this feature, eventually including one or multiple use case scenarios/examples?
2. Is there a way to see, in MS Fabric, where is the 'notebook resource'/custom library landing, after being exported from VS code? Will the 'notebook resource'/custom library be landing in the Files section of the data lake associated with the notebook, for instance, or is it somewhere else?
3. Related to VS Code and MS Fabric integration, and because I like to be an efficient programmer I find a bit clunky/not practical to keep pushing all those import/export buttons. Is it planned to enable some sort of automatic import/export of notebooks and their resources into or from MS Fabric ?

Again, thanks for the input.

[v-nikhilan-msft]

Hi @SergioTorrinha 

For the question 2 :
First step, click "open notebook button" in the tree view:

VScode will open a folder like this:

All the resource files are in the built in folder here:

We can right click the item and review it in file explorer to get it's address.

For questions 1 and 3 currently, there is no plan. I request you please share the feedback on our feedback channel . Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this helps. Please let me know if you have any further queries.

[SergioTorrinha · Accepted solution]

Hi @v-nikhilan-msft !

Thanks for the input.

Regarding question2:
So, that means there is no where, in MS Fabric workspace to see where the builtin folder is ? Perhaps this could be improved in future I would say, for the sake of integrity, clearness and completness.

Regarding questions 1 and 3:

I already did what you are sugesting 2 days ago. Here's the link for it, if someone would like to review/improve the idea and, eventually, give an up vote:
https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=8cf87c47-99a4-ee11-92bd-6045bdbdea69

Thank you

[v-nikhilan-msft]

Hi @SergioTorrinha 
Thanks for the feedback.

[v-nikhilan-msft]

Hi @SergioTorrinha 
Thanks for using Fabric Community.

1. Customized Libraries and Wheel Files:

Building wheel files every time you make adjustments can be quite tedious. Here are some alternatives:

• Pip-installable Packages: If your library is simple enough, consider packaging it as a regular Python package and publishing it on a private PyPI repository within your organization. This allows easy installation with pip install and avoids rebuilding wheels.
• Local Imports: For quick testing and development, you can import your library directly from the source directory without building a wheel. This speeds up your workflow but is limited to the development environment.
• VS Code Integration: You can use VS Code extensions like "Python Project Manager" or "Kite" to manage dependencies and automatically install modules from source directories within your workspace. This provides a more IDE-integrated approach.

2. Notebook Resources:

Notebook resources are indeed a new feature in Microsoft Fabric. They allow you to reference external Python modules directly within your notebook without needing to install them through traditional methods like wheels or pip.

To use this functionality you need to install the Synapse Visual Studio Code extension. Please refer to this link and install all the prerequisites : Link1
The Synapse Visual Studio Code extension supports a pro-developer experience for exploring Microsoft Fabric lakehouses, and authoring Fabric notebooks and Spark job definitions.

3. Microsoft Fabric Roadmap:

Microsoft Fabric is continuously evolving, and the team is always working on improving functionalities like notebooks, python environments, and custom libraries. The specific roadmap for these areas isn't publicly available, but you can find general updates and announcements through the Microsoft Fabric blog and release plan documentation.

Hope this helps. Please let us know if you have any further questions.

[v-nikhilan-msft]

Hi @SergioTorrinha 
We haven’t heard from you on the last response and was just checking back to see if you got some insights regarding your query.
Otherwise, will respond back with the more details and we will try to help.
Thanks