Forum Discussion
Reference current workspace with notebook without Spark
Hello,
I have a dev workspace with a notebook which will read data from a table in a lakehouse from the same dev workspace.
Later I will publish the objects to the test workspace and I want the notebook to reference the table in the lakehouse in the test workspace, automatically, without having to manually change a hard-coded path.
Here is how I can do it WITH spark:
from notebookutils import mssparkutils
this_workspace_id = mssparkutils.lakehouse.get('lakehouse')['workspaceId']
this_lakehouse_id = mssparkutils.lakehouse.get('lakehouse')['id']
table_path = f'abfss://{this_workspace_id}@onelake.dfs.fabric.microsoft.com/{this_lakehouse_id}/Tables/dbo/table'
spark.read.format("delta").option("startingVersion", "latest").load(table_path)
But I want to do it without starting a Spark session. Without a Spark session you can't import mssparkutils from notebookutils.
As far as I can tell it's impossible to write data to the Tables section of a datalake without starting a Spark session, so this approach will not work.
10 Replies
Hi DCELL ,
One solution I’d recommend is leveraging Fabric Pipelines for orchestration to retrieve the current workspace and pass it as a parameter when calling the notebook. This allows your notebook to dynamically reference the appropriate Lakehouse without hardcoding any paths.
You can then deploy artifacts from dev to test using Fabric Deployment Pipelines. Since both your pipeline and notebook are parameterized, they’ll automatically adapt to the target environment during deployment.
Also, because the workspace context is retrieved by the pipeline, a Spark session will only be initiated when the notebook runs, not before. This avoids the need to import mssparkutils outside of a Spark session.
Here is a blog I posted that shows a similar use-case and tutorial, however with Warehouses and Stored Procedures: https://discoveringallthingsanalytics.com/fabric-deployment-pipelines-guide-dynamic-warehouse-connections-in-microsoft-fabric-pipelines/If this helped, please mark it as the solution so others can benefit too. And if you found it useful, kudos are always appreciated.
Thanks,Samson
It's half a solution because I could read the data with Spark, write some code, and when it's ready then switch to pd.read_parquet and add parameterization with the pipeline.
But ideally I want to be able to get the lakehouse & workspace reference within the notebook itself because it also allows me to do some development in a non-Spark notebook and it won't by blocked (due to the Spark session limit) by another Spark-enabled notebook which is already running.
Hi DCELL ,Thanks for reaching out to the Microsoft fabric community forum
Thanks for your prompt response
DCELL ,
You're right getting the Lakehouse/workspace info inside a non-Spark notebook is currently not directly supported like it is with mssparkutils in Spark.
However, you can still achieve dynamic, environment-aware notebooks by parameterizing them and using Fabric Pipelines to inject those values at runtime this way, you avoid hardcoding, and your notebook stays Spark-free.
As a lightweight alternative, you could also read a small config.json file from the Lakehouse Files/ area that contains workspace/Lakehouse metadata this works fine in pandas’ notebooks.
So, while the feature isn’t natively exposed in non-Spark notebooks (yet), it’s still possible to design a dynamic, scalable workflow without requiring Spark sessions.
NotebookUtils (former MSSparkUtils) for Fabric - Microsoft Fabric | Microsoft Learn
The Microsoft Fabric deployment pipelines process - Microsoft Fabric | Microsoft Learn
If this post helped resolve your issue, please consider giving it Kudos and marking it as the Accepted Solution. This not only acknowledges the support provided but also helps other community members find relevant solutions more easily.
We appreciate your engagement and thank you for being an active part of the community.
Best regards,
LakshmiNarayana.
