Forum Discussion
Best Practices (?) for invoking Notebooks through Notebooks
Dear community,
we are trying to make "post deployment scripts" work by invoking child notebooks through a parent notebook.
ParentNotebook (default lakehouse: lh1)
--- invokes ChildNotebook1 (default lakehouse: lh1)
--- invokes ChildNotebook2 (default lakehouse: lh2)
Use case: Rename a column in lh1.
- Parent notebook drops table tbl1 in lakehouse lh1 to avoid schema collision upon recalculation of the table.
- Parent notebook then invokes ChildNotebook1 that generates table tbl1 in lh1 (with the new column name).
Now the catch. We have another lakehouse lh2 for which we want to use the same post deployment notebook. The child generates a table tbl2. BUT, since
default lakehouse of ParentNotebook = lh1!= lh2 = default lakehouse of ChildNotebook2
I receive an error
AnalysisException
Couldn't find a catalog to handle the identifier lh2.dbo.tbl2.
Issue: When starting a Spark session, the sessions metastore is initialized with the default lakehouse's data catalog and cannot be altered down the line. I assume that this is due to data security.
Current workaround:
- Run the ParentNotebook once with default lakehouse = lh1.
- Re-run the notebook with default lakehouse = lh2.
but this is somewhat cumbersome and manual. Are there any best practices?
NOTE: changes in the tables lh1 unfortunately affect changes in the tables in lh2, which is why splitting post deployment into two scripts is also not ideal.
An online search has not yielded any satisfying results. Please feel free to comment and help us out.
Thank you very much!
Tobias
4 Replies
- v-achippa
Community Support
Hi T-Eichinger,
Thank you for the response and confirming that the issue is resolved now. Thank you for being part of Microsoft Fabric Community.
Thanks and regards,
Anjan Kumar Chippa - T-EichingerNew Member
Dear GilbertQ ,
thank you very much!
I just tested notebook execution with ABFSS path references and it worked!
Best wishes from Berlin,
Tobias
v-achippa , we can close this thread :)
Hi T-Eichinger,
Thank you for reaching out to Microsoft Fabric Community.
Thank you GilbertQ for the prompt response.
As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user for the issue worked? or let us know if you need any further assistance.
Thanks and regards,
Anjan Kumar Chippa- GilbertQ
Super User
Hi T-Eichinger
You can use a DAG to achieve what you are looking to do.
https://databear.com/dag-orchestration-in-microsoft-fabric/
What I would also recommend is that you do not need to attach a Lakehouse to a notebook, if you use the ABFSS path you then pass through the workspace ID (GUID) and lakehouse ID (GUID).
Likewise, if you are using a pipeline, you can pass through parameters from the pipeline to the Notebook to make it dynamic?