Updating lakehouse reference in Pipeline

[bensimpsy]

Hi All,

I currently have 3 environments setup (Dev, Test & Prod) and currently configuring pipelines to move through each stage.

I have a set of notebooks which retrieve data from a Bronze Lakehouse, manipulate and save to a Silver Lakehouse (and the same from Silver to Gold). 

When configuring a pipeline I have the option to update the Default Lakehouse, but not any other referenced Lakehouse. Resulting in the notebooks in my Test environment looking like this, where the default correctly references my Test Lakehouse but my other referenced lakehouse is still referencing my Dev Lakehouse.

I've tried manually updating the reference in the Test Environment, but any further deployments override this and swap it back.

Have I just missed something somewhere, or is there no way to update the other Lakehouse references that are not the default Lakehouse in a Pipeline?  Do I seriously have to reference any non default Lakehouse using the fully qualified path in the notebook?

Regards,

Ben

[michalikmaciek]

For the remaining lakehouses, try to mount them using mssparkutils. I think you can pass the ID as a parameter to the notebooks. https://learn.microsoft.com/pl-pl/fabric/data-engineering/microsoft-spark-utilities 

[bensimpsy]

Hi @Anonymous thanks for looking at this.

I did review that article, and my issue is somewhat related.  I dont have a problem with the default Lakehouse, only other linked Lakehouses.

I have separate environments for my notebooks/data pipelines for dev, test and prod as well as separate lakehouses for each dev, test and prod.  The idea being the dev workspace containing the code relates to the dev lakehouse, test referencing test etc.

Most of my notebooks follow the same process in that:

• Read data from default Lakehouse using a query such as:

df = spark.sql("SELECT * FROM Silver.TableName")​

• Perform some manipulation, joins whatever else.
• Save the data to the next layer/Lakehouse using something like the following:

df.write.format("delta").mode("overwrite").option("overwriteSchema", "true").saveAsTable("Gold.TableName")​

An example of what my connected Lakehouses for a notebook would look like this showing 2 connected lakehouses (in the above example the default being Silver and the other being Gold:

The above works perfectly fine when in my dev workspace and I can configure the deployment pipeline to update the workspace and lakehouse Id's of the default lakehouse (as per your linked article and screenshot) but there is nothing about how to update the references to any other lakehouses used by the notebook.  As a result, when deploying to my test environment I end up in a position where my default lakehouse has successfully moved over to my test lakehouse, but the other lakehouse is still referencing the lakehouse from my dev environment (per my original screenshot).

I was hoping to avoid having to use fully qualified paths to the other lakehouses referenced by my notebook? I am not really sure why there is a tool available as part of the deployment pipeline to update the default but not any other?

I am relatively new to this, so if this structure isn't best practice and there is a better solution I'd be grateful for some advice.

Thanks

[Anonymous]

Hi @bensimpsy ,

At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

[Anonymous]

Hi @bensimpsy ,

The best course of action is to open a support ticket and have our support team take a closer look at it.

Please reach out to our support team so they can do a more thorough investigation on why this it is happening: Link 

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Hope this helps. Please let us know if you have any other queries.

[Anonymous]

Hi @bensimpsy 

We haven’t heard from you on the last response and was just checking back to see if you got a chance to open a support ticket.

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thanks

[bensimpsy]

Hi @Anonymous 

I have raised a support ticket: Case 2405300010005431

I've also worked around this by referencing the fully qualified path (not ideal but needed to move forward).

Thanks

[Anonymous]

Hi @bensimpsy ,

Thanks for sharing the support ticket.

Please allow some time, so team can check and provide a resolution.

In case if you got a resolution please do share with the community as it can be helpful to others .

[Anonymous]

Hi @bensimpsy ,

Thanks for using Fabric Community.
I assume you are taking about Deployment Pipelines.
Did you got a chance to look into this?  deployment-pipelines/create-rules 

If your is query is different, can you please explain with little more context so that I can guide you better.

Hope this is helpful. Please do let me know incase of further queries.