Forum Discussion
Mounting NB on- the-fly
smpa01
Could you explain more in detail what you want to do with the data?
Moving data from one workspace to another?
There could be easier ways to accomplish this instead of attaching Lakehouse programmatically to Notebook.
Here is a solution which doesn't involve mounting or attaching Lakehouses to the Notebook:
Perhaps you can use temporary views to be able to work with Spark SQL without having a default Lakehouse for your Notebook.
In this example I have a Notebook without any default Lakehouse and without any mounted/attached lakehouses.
I create a dataframe to connect to data from a lakehouse using the abfss path, and create a temporary view based on that dataframe.
df = spark.read.load("abfss://<workspaceID>@onelake.dfs.fabric.microsoft.com/<lakehouseID>/Tables/<tableName>")
df.createOrReplaceTempView("MyView_Read")
I use Spark SQL to do some modifications to the temporary view, and save it as another temporary view.
%%sql
CREATE OR REPLACE TEMPORARY VIEW MyView_Write AS SELECT *, Current_Timestamp as LoadTime FROM MyView_Read;
I create a new dataframe from the new temporary new.
I then write this new dataframe to another lakehouse table in another workspace.
df_write = spark.sql("SELECT * FROM MyView_Write")
df_write.write.mode("append").save("<anotherWorkspaceID>@onelake.dfs.fabric.microsoft.com/<anotherLakehouseName>/Tables/<anotherTableName>")
This seems to work fine for me.