Forum Discussion

T-Eichinger's avatar
T-Eichinger
New Member
4 days ago

Best Practices (?) for invoking Notebooks through Notebooks

Dear community,

we are trying to make "post deployment scripts" work by invoking child notebooks through a parent notebook.

ParentNotebook (default lakehouse: lh1)

--- invokes ChildNotebook1 (default lakehouse: lh1)

--- invokes ChildNotebook2 (default lakehouse: lh2)

Use case: Rename a column in lh1. 

  1. Parent notebook drops table tbl1 in lakehouse lh1 to avoid schema collision upon recalculation of the table.
  2. Parent notebook then invokes ChildNotebook1 that generates table tbl1 in lh1 (with the new column name).

Now the catch. We have another lakehouse lh2 for which we want to use the same post deployment notebook. The child generates a table tbl2. BUT, since 

default lakehouse of ParentNotebook = lh1!= lh2 = default lakehouse of ChildNotebook2

I receive an error

AnalysisException

Couldn't find a catalog to handle the identifier lh2.dbo.tbl2.

Issue: When starting a Spark session, the sessions metastore is initialized with the default lakehouse's data catalog and cannot be altered down the line. I assume that this is due to data security.

Current workaround:

  1. Run the ParentNotebook once with default lakehouse = lh1. 
  2. Re-run the notebook with default lakehouse = lh2.

but this is somewhat cumbersome and manual. Are there any best practices?

NOTE: changes in the tables lh1 unfortunately affect changes in the tables in lh2, which is why splitting post deployment into two scripts is also not ideal. 

An online search has not yielded any satisfying results. Please feel free to comment and help us out.

 

Thank you very much!

Tobias

4 Replies

  • v-achippa's avatar
    v-achippa
    Icon for Community Support rankCommunity Support

    Hi T-Eichinger​,

    Thank you for the response and confirming that the issue is resolved now. Thank you for being part of Microsoft Fabric Community.

    Thanks and regards,
    Anjan Kumar Chippa

  • Dear GilbertQ​ ,

    thank you very much!

    I just tested notebook execution with ABFSS path references and it worked!

    Best wishes from Berlin,

    Tobias

    v-achippa​ , we can close this thread :)

  • v-achippa's avatar
    v-achippa
    Icon for Community Support rankCommunity Support

    Hi T-Eichinger​,

    Thank you for reaching out to Microsoft Fabric Community.

    Thank you GilbertQ​ for the prompt response.

    As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user for the issue worked?  or let us know if you need any further assistance.

    Thanks and regards,
    Anjan Kumar Chippa

  • Hi T-Eichinger​ 

    You can use a DAG to achieve what you are looking to do.

    https://databear.com/dag-orchestration-in-microsoft-fabric/ 

    What I would also recommend is that you do not need to attach a Lakehouse to a notebook, if you use the ABFSS path you then pass through the workspace ID (GUID) and lakehouse ID (GUID).

    Likewise, if you are using a pipeline, you can pass through parameters from the pipeline to the Notebook to make it dynamic?