Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Enhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.

Reply
mikkeldema
New Member

NotebookUtils run makes the runned notebook inherit default lakehouse from caller

Hey

 

I have a problem with using NotebookUtils. My design is that I have two workspaces wilt following utilities:

-BRONZE WORKSPACE-

Notebook: Ochrestrator - default lakehouse is Bronze

Notebook: Process Bronze - default lakehouse is Bronze

Datalake: Bronze

 

-SILVER WORKSPACE-

Notebook: Process Silver - default lakehouse is Silver

Datalake: Silver

 

From Orchestrator i make these two notebook runs, but when executing Silver it seems like the default lakehouse is inherited from the calling notebook, because when I print the tables in the default lakehouse from silver it shows the bronze tables.

 

            # Run Bronze Event Processor
            processEventBronze = notebookutils.notebook.run(
                "Bronze Event Processor",
                300,
                {
                    "useRootDefaultLakehouse": True,
                    "entityType": event['eventType'],
                    "schema": entitySourceTransformed,
                    "entityName": entityName,
                    "entityTableName": bronzeTableName,
                    "entityKey": entityKey,
                    "correlationId": correlationId,
                    "payload": payload
                },
                bronzeTargetWorkspace
            )
            print(processEventBronze)

            # Run Silver Event Processor
            processEventSilver = notebookutils.notebook.run(
                "Silver Event Processor",
                300,
                {
                    "useRootDefaultLakehouse": True,
                    "bronzeTableName": bronzeTableName,
                    "silverTableName": silverTableName,
                    "entityKey": entityKey,
                    "entityId": entityId,
                    "entityType": event['eventType']
                },
                silverTargetWorkspace
            )
 
Hope someone has an idea of why this is happening when I try to access the silver lakehouse data.
 
# Load the bronze and silver DataFrames
        bronze_df = spark.read.format("delta").load(f"abfss://{sourceWorkspace}@onelake.dfs.fabric.microsoft.com/{sourceLakehouse}/Tables/{bronzeTableName}").filter(col(entityKey) == entityId)
        silver_df = spark.read.format("delta").load(f"Tables/{silverTableName}").filter(col("source_id") == entityId)
1 ACCEPTED SOLUTION
spencer_sa
Super User
Super User

I didn't know that .run keeps the same default lakehouse, but I can see it makes sense.  The default lakehouse is set at Spark Session start (you can parameterise it though)
.run doesn't create a new spark session, but reuses the old one ("The notebook being referenced runs on the Spark pool of the notebook that calls this function.") from here;
https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities

What we do is explicitly use the ABFSS path rather than default lakehouses.  (we also seperate the Notebooks/Pipelines into a separate workspace completely so have to use ABFSS paths to specify lakehouses.)

So df.read.format('delta').load('abfss://<silverworkspace>@onelake.dfs.fabric.microsoft.com/<silverlakehouse>/Tables/...')

View solution in original post

1 REPLY 1
spencer_sa
Super User
Super User

I didn't know that .run keeps the same default lakehouse, but I can see it makes sense.  The default lakehouse is set at Spark Session start (you can parameterise it though)
.run doesn't create a new spark session, but reuses the old one ("The notebook being referenced runs on the Spark pool of the notebook that calls this function.") from here;
https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities

What we do is explicitly use the ABFSS path rather than default lakehouses.  (we also seperate the Notebooks/Pipelines into a separate workspace completely so have to use ABFSS paths to specify lakehouses.)

So df.read.format('delta').load('abfss://<silverworkspace>@onelake.dfs.fabric.microsoft.com/<silverlakehouse>/Tables/...')

Helpful resources

Announcements
July 2025 community update carousel

Fabric Community Update - July 2025

Find out what's new and trending in the Fabric community.

June FBC25 Carousel

Fabric Monthly Update - June 2025

Check out the June 2025 Fabric update to learn about new features.