Forum Discussion

T-Eichinger's avatar
T-Eichinger
New Member
1 day ago

Best Practices (?) for invoking Notebooks through Notebooks

Dear community,

we are trying to make "post deployment scripts" work by invoking child notebooks through a parent notebook.

ParentNotebook (default lakehouse: lh1)

--- invokes ChildNotebook1 (default lakehouse: lh1)

--- invokes ChildNotebook2 (default lakehouse: lh2)

Use case: Rename a column in lh1. 

  1. Parent notebook drops table tbl1 in lakehouse lh1 to avoid schema collision upon recalculation of the table.
  2. Parent notebook then invokes ChildNotebook1 that generates table tbl1 in lh1 (with the new column name).

Now the catch. We have another lakehouse lh2 for which we want to use the same post deployment notebook. The child generates a table tbl2. BUT, since 

default lakehouse of ParentNotebook = lh1!= lh2 = default lakehouse of ChildNotebook2

I receive an error

AnalysisException

Couldn't find a catalog to handle the identifier lh2.dbo.tbl2.

Issue: When starting a Spark session, the sessions metastore is initialized with the default lakehouse's data catalog and cannot be altered down the line. I assume that this is due to data security.

Current workaround:

  1. Run the ParentNotebook once with default lakehouse = lh1. 
  2. Re-run the notebook with default lakehouse = lh2.

but this is somewhat cumbersome and manual. Are there any best practices?

NOTE: changes in the tables lh1 unfortunately affect changes in the tables in lh2, which is why splitting post deployment into two scripts is also not ideal. 

An online search has not yielded any satisfying results. Please feel free to comment and help us out.

 

Thank you very much!

Tobias

1 Reply

  • Hi T-Eichinger​ 

    You can use a DAG to achieve what you are looking to do.

    https://databear.com/dag-orchestration-in-microsoft-fabric/ 

    What I would also recommend is that you do not need to attach a Lakehouse to a notebook, if you use the ABFSS path you then pass through the workspace ID (GUID) and lakehouse ID (GUID).

    Likewise, if you are using a pipeline, you can pass through parameters from the pipeline to the Notebook to make it dynamic?