NotebookUtils run makes the runned notebook inherit default lakehouse from caller

Question

Hey&nbsp;I have a problem with using NotebookUtils. My design is that I have two workspaces wilt following utilities:-BRONZE WORKSPACE-Notebook: Ochrestrator - default lakehouse is BronzeNotebook: Process Bronze - default lakehouse is BronzeDatalake: Bronze&nbsp;-SILVER WORKSPACE-Notebook: Process Silver - default lakehouse is SilverDatalake: Silver&nbsp;From Orchestrator i make these two notebook runs, but when executing Silver it seems like the default lakehouse is inherited from the calling notebook, because when I print the tables in the default lakehouse from silver it shows the bronze tables.&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; # Run Bronze Event Processor&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; processEventBronze = notebookutils.notebook.run(&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "Bronze Event Processor", &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 300, &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "useRootDefaultLakehouse": True,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "entityType": event['eventType'],&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "schema": entitySourceTransformed,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "entityName": entityName,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "entityTableName": bronzeTableName,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "entityKey": entityKey,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "correlationId": correlationId,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "payload": payload&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }, &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; bronzeTargetWorkspace&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; )&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; print(processEventBronze)&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; # Run Silver Event Processor&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; processEventSilver = notebookutils.notebook.run(&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "Silver Event Processor", &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; 300, &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; {&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "useRootDefaultLakehouse": True,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "bronzeTableName": bronzeTableName,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "silverTableName": silverTableName,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "entityKey": entityKey,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "entityId": entityId,&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "entityType": event['eventType']&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }, &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; silverTargetWorkspace&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; )&nbsp;Hope someone has an idea of why this is happening when I try to access the silver lakehouse data.&nbsp;# Load the bronze and silver DataFrames&nbsp; &nbsp; &nbsp; &nbsp; bronze_df = spark.read.format("delta").load(f"abfss://{sourceWorkspace}@onelake.dfs.fabric.microsoft.com/{sourceLakehouse}/Tables/{bronzeTableName}").filter(col(entityKey) == entityId)&nbsp; &nbsp; &nbsp; &nbsp; silver_df = spark.read.format("delta").load(f"Tables/{silverTableName}").filter(col("source_id") == entityId)

spencer_sa · Accepted Answer

I didn't know that .run keeps the same default lakehouse, but I can see it makes sense. The default lakehouse is set at Spark Session start (you can parameterise it though)
.run doesn't create a new spark session, but reuses the old one ("The notebook being referenced runs on the Spark pool of the notebook that calls this function.") from here;
https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities

What we do is explicitly use the ABFSS path rather than default lakehouses. (we also seperate the Notebooks/Pipelines into a separate workspace completely so have to use ABFSS paths to specify lakehouses.)

So df.read.format('delta').load('abfss://<silverworkspace>@onelake.dfs.fabric.microsoft.com/<silverlakehouse>/Tables/...')

Forum Discussion

NotebookUtils run makes the runned notebook inherit default lakehouse from caller

1 Reply

Related content

Change default Lakehouse with notebookutils

REST API vs. Notebookutils Lakehouse Update Issue

Runtime Lakehouse Switch between multiple notebooks

Notebook default lakehouse error

Attach multiple lakehouse to a single notebook.