Updating lakehouse reference in Pipeline

Hi Anonymous thanks for looking at this.

I did review that article, and my issue is somewhat related.  I dont have a problem with the default Lakehouse, only other linked Lakehouses.

I have separate environments for my notebooks/data pipelines for dev, test and prod as well as separate lakehouses for each dev, test and prod.  The idea being the dev workspace containing the code relates to the dev lakehouse, test referencing test etc.

Most of my notebooks follow the same process in that:

• Read data from default Lakehouse using a query such as:

df = spark.sql("SELECT * FROM Silver.TableName")​

• Perform some manipulation, joins whatever else.
• Save the data to the next layer/Lakehouse using something like the following:

df.write.format("delta").mode("overwrite").option("overwriteSchema", "true").saveAsTable("Gold.TableName")​

An example of what my connected Lakehouses for a notebook would look like this showing 2 connected lakehouses (in the above example the default being Silver and the other being Gold:

The above works perfectly fine when in my dev workspace and I can configure the deployment pipeline to update the workspace and lakehouse Id's of the default lakehouse (as per your linked article and screenshot) but there is nothing about how to update the references to any other lakehouses used by the notebook.  As a result, when deploying to my test environment I end up in a position where my default lakehouse has successfully moved over to my test lakehouse, but the other lakehouse is still referencing the lakehouse from my dev environment (per my original screenshot).

I was hoping to avoid having to use fully qualified paths to the other lakehouses referenced by my notebook? I am not really sure why there is a tool available as part of the deployment pipeline to update the default but not any other?

I am relatively new to this, so if this structure isn't best practice and there is a better solution I'd be grateful for some advice.

Thanks