Remote and persistent Jupyter kernel in Microsoft Fabric

Hi roger_fabric ,

Thank you for reaching out to the Microsoft Fabric Community Forum.

Currently, Microsoft Fabric’s Spark compute nodes run in a secure, shared environment. That means when you use a PySpark kernel, it can’t directly access code stored only on your local machine. Even though “Open in VS Code” gives you a live connection to the kernel, your laptop and the Fabric cluster still have separate file systems. So, something like from src.utils import my_model won’t work unless you’ve uploaded that code to Fabric first.

For quick and easy development, it’s best to keep your project in Git, open it in VS Code with the Fabric extension, and use the Publish button. This lets you sync just the updated files to the notebook’s Resources folder, making them instantly available to the remote kernel no need to build a wheel.

When your code is ready to be reused across notebooks or pipelines, you can package it as a wheel (or just a set of .py files) and add it as a custom library in a Fabric Environment. Attaching that environment will automatically load your package every time a Spark session starts. If you’re just experimenting, you can also drag and drop a wheel file into the notebook and run %pip install to use it right away.

Since Fabric clusters are managed cloud resources, things like SSH access or mounting drives aren’t currently supported. However, there’s already a community request open for a “Spark-Connect-style” live file sync feel free to vote on it or else create an idea in the idea forum using the below link : Fabric Ideas - Microsoft Fabric Community

In the meantime, the recommended workflow is:

• Develop locally and use Git for version control
• Publish small changes while iterating and package your code when it’s stable

Hope this helps. Please reach out for further assistance.

If this post helps, then please consider to Accept as the solution to help the other members find it more quickly and a kudos would be appreciated.

Thank you.