Runtime Lakehouse Switch between multiple notebooks

[hareshparab-408]

Hi,
I do have workspace: ws-common, ws1, ws2 and lakehouse: lh_common, lh_1, lh_2.
consider ws-common is common so i am trying to keep it like this where there is common pipeline and common code. 
and if i have to process clients data i.e (ws1,lh_1), (ws2,lh_2)  i will trigger 2 instace of common pipeline where i will read all clients configs from lh_common and then each instance will run common notebooks and process clients data.

so i have created one notebook where i usually attach defaultlakehouse lh_common and read the config data such as which notebook to process (notebook will be only in common_wokspaces, no notebooks are created in client workspaces). from the current notebook i read config and call child notebooks to process now in order to process i need to attach lakehouse of that particular client but it should be dynamic. currently i am not able to do that  

[Lodha_Jaydeep · Accepted solution]

Hi @hareshparab-408,
You can try below dynamic options also to achive your desired workaround.

1) Don't switch attachments; use full onelake paths

Keep your lh_common attached for the config and use dynamic paths to read/write for the client data for example, 

client_id = "1" # from config table
lh_path = f"abfss://{workspace_id}@onelake.dfs.fabric.microsoft.com/{lakehouse_id}/Tables/YourTable"
df = spark.read.format("delta").load(lh_path)

Or use the SQL endpoint / Spark.SQL

SELECT * FROM 'ws1'.'lh_1'.'dbo'.'YourTable'

2) Create parameterized pipeline for each run.

Create commom Data pipeline in the workspace you want e.g ws-common

• Read config from the lh_common
• For client data processing use same notebook activity with parametrs such as workspace_name, lakehouse_name,table_path
• Notebook will pick the update parameters value and execute by build path dynamically (As same as above option)

I hope this is what you need, please consider as an accepted solution if helps or give some kudos.

Thanks

[v-moharafi-msft]

Hi   @hareshparab-408  ,  

Could you please confirm if the issue has been resolved? If not, feel free to reach out if you have any further questions.

Your update would be helpful for other members who may face a similar issue.

Best Regards,

Abdul Rafi

[v-moharafi-msft]

Hi  @hareshparab-408   ,

Thank you for reaching out to Microsoft Fabric Community and Thanks to @tayloramy, @GilbertQ, @Lodha_Jaydeep    and @Olufemi7     for Sharing valuable insights.

Just wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. 

 Best Regards,

Abdul Rafi.

[tayloramy · Accepted solution]

Hi @hareshparab-408, 

Another option is to use the 4 part identifier to reference the lakehouse, this should work regardless of the lakehouse context you have (as long as you have a context) 

use `workspace name`.lakehouse.schema.table
use `` if your workspace name has spaces. 

 

[Lodha_Jaydeep · Accepted solution]

Hi @hareshparab-408,
You can try below dynamic options also to achive your desired workaround.

1) Don't switch attachments; use full onelake paths

Keep your lh_common attached for the config and use dynamic paths to read/write for the client data for example, 

client_id = "1" # from config table
lh_path = f"abfss://{workspace_id}@onelake.dfs.fabric.microsoft.com/{lakehouse_id}/Tables/YourTable"
df = spark.read.format("delta").load(lh_path)

Or use the SQL endpoint / Spark.SQL

SELECT * FROM 'ws1'.'lh_1'.'dbo'.'YourTable'

2) Create parameterized pipeline for each run.

Create commom Data pipeline in the workspace you want e.g ws-common

• Read config from the lh_common
• For client data processing use same notebook activity with parametrs such as workspace_name, lakehouse_name,table_path
• Notebook will pick the update parameters value and execute by build path dynamically (As same as above option)

I hope this is what you need, please consider as an accepted solution if helps or give some kudos.

Thanks

[GilbertQ]

Hi @hareshparab-408 

There is also another option where you don't have to attach a Lakehouse to a notebook, but when you do that you then have to specify the workspace and lake house goods which can be parameter driven in order to write successfully to the lake house tables using the ABFSS path.

# Using GUIDs (most reliable)
abfss_path = "abfss://<workspace_id>@onelake.dfs.fabric.microsoft.com/<lakehouse_id>/Tables/my_table"

# Or using names (if no special chars/spaces)
abfss_path = "abfss://MyWorkspace@onelake.dfs.fabric.microsoft.com/MyLakehouse.Lakehouse/Tables/my_table"

[Olufemi7]

Hello @hareshparab-408,
Fabric binds a notebook to a defaultLakehouse at session start, so the Lakehouse context cannot be changed during execution.
Keep lh_common attached for config.
Pass client_workspace and client_lakehouse as parameters for each run, then use those parameters to read and write client data via OneLake paths.
Docs:
Develop, execute, and manage Microsoft Fabric notebooks