Forum Discussion

KrishnaMoola's avatar
KrishnaMoola
Regular Visitor
7 months ago
Solved

Attach multiple lakehouse to a single notebook.

I have a notebook and have attached to a lakehouse. The default lakehouse is called metadata, where I keep details about the job runs. Now, I have created Silver lakehouse and within a notebook at each step I need to insert when the process started n ended and also write to Silver tables. I am getting an error "[SCHEMA_NOT_FOUND] The schema `silver` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a catalog, verify the current_schema() output, or qualify the name with the correct catalog."

 

  • KrishnaMoola's avatar
    KrishnaMoola
    7 months ago

    Hi V-yubandi-msft , sorry for coming back late on this, I was on leave and not feeling well. The issue is resolved now. Was able to attach multiple lakehouses to the notebook which resolved the issue

15 Replies

  • KrishnaMoola you can add multiple lakehouses to a notebook. 

    In your notebook, click the add data items button: 

     

     

    then you can choose an existing datasource and add another lakehouse: 

     

     

    THe default lakehouse is still the one with the pin beside it, so to reference the other lakehouse you need to use the full identifier: 
    GatewayLogs.dbo.table1, not just dbo.table1 

     

    If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

    • KrishnaMoola's avatar
      KrishnaMoola
      Regular Visitor

      I have given GatewayLogs.dbo.table1 but still no luck, so have created a shortcut to the metadata lakehouse and this works. I am not sure if this is correct way to do

      • V-yubandi-msft's avatar
        V-yubandi-msft
        Community Support

        Hi KrishnaMoola ,

        Thanks for the update. Creating a shortcut will work because it exposes the Silver table inside your default lakehouse, so Spark is able to resolve it. However, this is more of a workaround.

         

        The recommended approach is to attach the Silver lakehouse as a Data item in the notebook. Once both lakehouses are attached, the tables in Silver should be recognised directly using the standard namespace format.

        So your shortcut isn’t incorrect but attaching the Silver lakehouse as a data item is cleaner and keeps the table where it belongs.

         

        If you still can’t reference the tables after attaching the Silver lakehouse, please let us know and we’ll help investigate further

  • You have bronze as default lakehouse, you can use the abfs path to get on silver lakehouse schema and inster data.

  • This implies a naming convention error. What notation do you get if you drag and drop the tables into your notebook?

  • Hi KrishnaMoola ,

    Are both of the lakehouses schema enabled?

     

    May be this will help?

     

    Lakehouse schemas in notebook
    When you look at a schema enabled lakehouse in the notebook object explorer, you see tables are in schemas. You can drag and drop table into a code cell and get a code snippet that refers to the schema where the table is located. Use this namespace to refer to tables in your code: "workspace.lakehouse.schema.table". If you leave out any of the elements, the executor uses default setting. For example, if you only give table name, it uses default schema (dbo) from default lakehouse for the notebook.

     

     

     

    Referencing nonschema lakehouses
    When you set a schema-enabled lakehouse or no lakehouse as the default in your notebook, Spark code uses schema-enabled referencing for tables. However, you can still access lakehouses without enabled schemas within the same code by referencing them as "lakehouse.table."

    Additionally, it’s possible to join tables from different types of lakehouses.

    SQL
    SELECT *
    FROM workspace.schemalh.schema.table as schematable
    INNER JOIN workspace.nonschemalh.table as nonschematable
    ON schematable.id = nonschematable.id;


    To help transition and refactor existing code to schema-enabled lakehouses, four-part naming is supported for lakehouses without schemas. You can reference tables as "workspace.lakehouse.dbo.table", where "dbo" serves as the schema name—even though it doesn’t exist yet in a lakehouse that isn’t schema-enabled. This approach lets you update your code with no downtime before enabling schema support in your lakehouses.

    • KrishnaMoola's avatar
      KrishnaMoola
      Regular Visitor

      Hi there, I tried to attach my notebook to Silver lakehouse but still the tables in silver notebook are not recognised. I tried giving the workspace.lakehouse.schema.table but no luck.

      Is there any other way ?

  • I created a shortcut to the table in Silver lakehouse but not sure if this is the correct approach, can anyone suggest on this ?

  • Hi KrishnaMoola ,

    we wanted to check in as we haven't heard back from you. Did our solution work for you? If you need any more help, please don't hesitate to ask. Your feedback is very important to us. We hope to hear from you soon.

  • Hi KrishnaMoola 

    In Fabric, the spark catelog only keeps the default lakehosue tables and schemas. Thus you are not able to access the other lakehouse(silver lakehouse) tables. The most appropriate option would be to use ABFS path. 
    Other than that if you are using spark.sql  you can use first convert the dataframe in VIEW and then perform sql operations. 


    Hope this helps. Let me know if you have any futher question , we can discuss !!

    • KrishnaMoola's avatar
      KrishnaMoola
      Regular Visitor

      Hi V-yubandi-msft , sorry for coming back late on this, I was on leave and not feeling well. The issue is resolved now. Was able to attach multiple lakehouses to the notebook which resolved the issue

      • V-yubandi-msft's avatar
        V-yubandi-msft
        Community Support

        Thank you for the update, and no worries at all.Glad to hear the issue is resolved and that attaching multiple lakehouses to the notebook addressed the problem.

         

        Please feel free to reach out if you need any further assistance.

         

        Thank You.

  • Because only the default attached Lakehouse is used automatically in a Fabric notebook, Spark can’t find the silver schema in the other Lakehouse , use fully qualified table name (LakehouseName.silver.table) or switch the catalog/schema explicitly.