Power BI is turning 10! Tune in for a special live episode on July 24 with behind-the-scenes stories, product evolution highlights, and a sneak peek at what’s in store for the future.
Save the dateEnhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.
Hey
I have a problem with using NotebookUtils. My design is that I have two workspaces wilt following utilities:
-BRONZE WORKSPACE-
Notebook: Ochrestrator - default lakehouse is Bronze
Notebook: Process Bronze - default lakehouse is Bronze
Datalake: Bronze
-SILVER WORKSPACE-
Notebook: Process Silver - default lakehouse is Silver
Datalake: Silver
From Orchestrator i make these two notebook runs, but when executing Silver it seems like the default lakehouse is inherited from the calling notebook, because when I print the tables in the default lakehouse from silver it shows the bronze tables.
Solved! Go to Solution.
I didn't know that .run keeps the same default lakehouse, but I can see it makes sense. The default lakehouse is set at Spark Session start (you can parameterise it though)
.run doesn't create a new spark session, but reuses the old one ("The notebook being referenced runs on the Spark pool of the notebook that calls this function.") from here;
https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities
What we do is explicitly use the ABFSS path rather than default lakehouses. (we also seperate the Notebooks/Pipelines into a separate workspace completely so have to use ABFSS paths to specify lakehouses.)
So df.read.format('delta').load('abfss://<silverworkspace>@onelake.dfs.fabric.microsoft.com/<silverlakehouse>/Tables/...')
I didn't know that .run keeps the same default lakehouse, but I can see it makes sense. The default lakehouse is set at Spark Session start (you can parameterise it though)
.run doesn't create a new spark session, but reuses the old one ("The notebook being referenced runs on the Spark pool of the notebook that calls this function.") from here;
https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-utilities
What we do is explicitly use the ABFSS path rather than default lakehouses. (we also seperate the Notebooks/Pipelines into a separate workspace completely so have to use ABFSS paths to specify lakehouses.)
So df.read.format('delta').load('abfss://<silverworkspace>@onelake.dfs.fabric.microsoft.com/<silverlakehouse>/Tables/...')
This is your chance to engage directly with the engineering team behind Fabric and Power BI. Share your experiences and shape the future.
User | Count |
---|---|
5 | |
4 | |
2 | |
2 | |
2 |