Forum Discussion
Runtime Lakehouse Switch between multiple notebooks
- 2 months ago
Hi hareshparab-408,
You can try below dynamic options also to achive your desired workaround.1) Don't switch attachments; use full onelake paths
Keep your lh_common attached for the config and use dynamic paths to read/write for the client data for example,
client_id = "1" # from config table
lh_path = f"abfss://{workspace_id}@onelake.dfs.fabric.microsoft.com/{lakehouse_id}/Tables/YourTable"
df = spark.read.format("delta").load(lh_path)Or use the SQL endpoint / Spark.SQL
SELECT * FROM 'ws1'.'lh_1'.'dbo'.'YourTable'
2) Create parameterized pipeline for each run.
Create commom Data pipeline in the workspace you want e.g ws-common
- Read config from the lh_common
- For client data processing use same notebook activity with parametrs such as workspace_name, lakehouse_name,table_path
- Notebook will pick the update parameters value and execute by build path dynamically (As same as above option)
I hope this is what you need, please consider as an accepted solution if helps or give some kudos.
Thanks
- 2 months ago
Hi hareshparab-408,
Another option is to use the 4 part identifier to reference the lakehouse, this should work regardless of the lakehouse context you have (as long as you have a context)
use `workspace name`.lakehouse.schema.table
use `` if your workspace name has spaces.
Hi hareshparab-408,
You can try below dynamic options also to achive your desired workaround.
1) Don't switch attachments; use full onelake paths
Keep your lh_common attached for the config and use dynamic paths to read/write for the client data for example,
client_id = "1" # from config table
lh_path = f"abfss://{workspace_id}@onelake.dfs.fabric.microsoft.com/{lakehouse_id}/Tables/YourTable"
df = spark.read.format("delta").load(lh_path)
Or use the SQL endpoint / Spark.SQL
SELECT * FROM 'ws1'.'lh_1'.'dbo'.'YourTable'
2) Create parameterized pipeline for each run.
Create commom Data pipeline in the workspace you want e.g ws-common
- Read config from the lh_common
- For client data processing use same notebook activity with parametrs such as workspace_name, lakehouse_name,table_path
- Notebook will pick the update parameters value and execute by build path dynamically (As same as above option)
I hope this is what you need, please consider as an accepted solution if helps or give some kudos.
Thanks