Forum Discussion

smpa01's avatar
smpa01
Icon for Community Champion rankCommunity Champion
2 years ago

Mounting NB on- the-fly

I am running some codes in a standalone notebook that is currently giving me the following error Spark SQL queries are only possible in the context of a lakehouse. Please attach a lakehouse to pro...
lakehouse
smpa01's avatar
smpa01
Icon for Community Champion rankCommunity Champion
2 years ago

This post describes how to mount a NB which is exactly similar to what I have provided (good so far)

but it does not execute a code in a Notebook manually attached to a Lakehouse would have successfully executed otherwise.

 

The objective of progrmatic mounting is to see if the succedding code executes correctly as it would have in a manually attached lakehouse.

frithjof_v's avatar
frithjof_v
Icon for Community Champion rankCommunity Champion
2 years ago

I see, I will do a test on my side if I find the time to do the test.

 

Btw I see you are using notebookutils whereas the blog post is using mssparkutils.

 

Will it work if you use mssparkutils instead of notebookutils?

  • smpa01's avatar
    smpa01
    Icon for Community Champion rankCommunity Champion
    2 years ago

    frithjof_v  i will check and let you know.

    • frithjof_v's avatar
      frithjof_v
      Icon for Community Champion rankCommunity Champion
      2 years ago

      I must admit I am not sure what it means to Attach or Mount a lakehouse to a Notebook. Is Attach and Mount the same thing? 

       

      Anyway, I am unsure if the mssparkutils mount method is able to make a lakehouse the default lakehouse for the Notebook.

      Okay, it can mount one or more lakehouses to the Notebook, which per my current understanding makes it possible to use relative directory paths (e.g. useful for os module and Pandas).

      However, if you want to use Spark SQL, I guess we would need to be able to mount a lakehouse as the default Lakehouse of the Notebook? I am not sure if the mssparkutils mount method can do that.

       

      It seems I am able to mount a Lakehouse as the default Lakehouse by using the solution to this forum thread: Solved: Re: How to set default Lakehouse in the notebook p... - Microsoft Fabric Community

       

      However I need to use this code (need to add the -f argument):

       

      %%configure -f
{
    "defaultLakehouse": {  
        "name": "<lakehouseName>",
        "id": "<lakehouseID>",
        "workspaceId": "<workspaceID>"
    }
}

       

       

      It means the Livy session will need to restart. I don't know what the Livy session is, however it seems all variables will get lost by doing that. 

       

      I also didn't find out how to insert the lakehouseName, lakehouseID and workspaceID as variables.
      So I had to hardcode those values in the JSON structure in the %%configure cell.
      I guess it should be possible to pass variables into the JSON structure, however at the moment I don't know how to do that.

       

      Other than that, it seemed to work, when I hardcoded the values.

      • frithjof_v's avatar
        frithjof_v
        Icon for Community Champion rankCommunity Champion
        2 years ago

        This worked for me in a Notebook which has no default lakehouse. The default lakehouse gets attached programmatically in the %%configure code cell.
        I hardcoded the values for lakehouseName, lakehouseID and workspaceID (as mentioned in previous comment).

         

        %%configure -f
{
    "defaultLakehouse": {  
        "name": "<lakehouseName>",
        "id": "<lakehouseID>",
        "workspaceId": "<workspaceID>"
    }
}

         

         

         

        %%sql
CREATE OR REPLACE TEMPORARY VIEW Dim_Product_temp_vw
AS 
SELECT *, CURRENT_TIMESTAMP AS load_timestamp 
FROM Dim_Product

         

         

         

        df = spark.sql("SELECT * FROM Dim_Product_temp_vw")

df.write.mode("append").saveAsTable("Dim_Product_2")

         

         

        However, if you want to switch to another default Lakehouse on-the-fly, then the temporary view will not be available after you switch to another default Lakehouse.

        Because the variables get lost when running the %%configure -f code cell.

         

        So I guess you would need to solve that using a workaround.