Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Be one of the first to start using Fabric Databases. View on-demand sessions with database experts and the Microsoft product team to learn just how easy it is to get started. Watch now

Reply
bensimpsy
Frequent Visitor

Updating lakehouse reference in Pipeline

Hi All,

 

I currently have 3 environments setup (Dev, Test & Prod) and currently configuring pipelines to move through each stage.

I have a set of notebooks which retrieve data from a Bronze Lakehouse, manipulate and save to a Silver Lakehouse (and the same from Silver to Gold). 

 

When configuring a pipeline I have the option to update the Default Lakehouse, but not any other referenced Lakehouse. Resulting in the notebooks in my Test environment looking like this, where the default correctly references my Test Lakehouse but my other referenced lakehouse is still referencing my Dev Lakehouse.

ZoomIt64_R0k6wUcI1I.png

I've tried manually updating the reference in the Test Environment, but any further deployments override this and swap it back.

 

Have I just missed something somewhere, or is there no way to update the other Lakehouse references that are not the default Lakehouse in a Pipeline?  Do I seriously have to reference any non default Lakehouse using the fully qualified path in the notebook?

 

Regards,

Ben

8 REPLIES 8
michalikmaciek
New Member

For the remaining lakehouses, try to mount them using mssparkutils. I think you can pass the ID as a parameter to the notebooks. https://learn.microsoft.com/pl-pl/fabric/data-engineering/microsoft-spark-utilities 

bensimpsy
Frequent Visitor

Hi @Anonymous thanks for looking at this.

I did review that article, and my issue is somewhat related.  I dont have a problem with the default Lakehouse, only other linked Lakehouses.

 

I have separate environments for my notebooks/data pipelines for dev, test and prod as well as separate lakehouses for each dev, test and prod.  The idea being the dev workspace containing the code relates to the dev lakehouse, test referencing test etc.

 

Most of my notebooks follow the same process in that:

  • Read data from default Lakehouse using a query such as:

 

df = spark.sql("SELECT * FROM Silver.TableName")​

 

  • Perform some manipulation, joins whatever else.
  • Save the data to the next layer/Lakehouse using something like the following:

 

df.write.format("delta").mode("overwrite").option("overwriteSchema", "true").saveAsTable("Gold.TableName")​

 

An example of what my connected Lakehouses for a notebook would look like this showing 2 connected lakehouses (in the above example the default being Silver and the other being Gold:

bensimpsy_0-1716547903122.png

 

 

The above works perfectly fine when in my dev workspace and I can configure the deployment pipeline to update the workspace and lakehouse Id's of the default lakehouse (as per your linked article and screenshot) but there is nothing about how to update the references to any other lakehouses used by the notebook.  As a result, when deploying to my test environment I end up in a position where my default lakehouse has successfully moved over to my test lakehouse, but the other lakehouse is still referencing the lakehouse from my dev environment (per my original screenshot).

 

I was hoping to avoid having to use fully qualified paths to the other lakehouses referenced by my notebook? I am not really sure why there is a tool available as part of the deployment pipeline to update the default but not any other?

 

I am relatively new to this, so if this structure isn't best practice and there is a better solution I'd be grateful for some advice.

 

Thanks

Anonymous
Not applicable

Hi @bensimpsy ,

At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

Anonymous
Not applicable

Hi @bensimpsy ,

The best course of action is to open a support ticket and have our support team take a closer look at it.

 

Please reach out to our support team so they can do a more thorough investigation on why this it is happening: Link 

 

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

 

Hope this helps. Please let us know if you have any other queries.

Anonymous
Not applicable

Hi @bensimpsy 

We haven’t heard from you on the last response and was just checking back to see if you got a chance to open a support ticket.

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thanks

Hi @Anonymous 

I have raised a support ticket: Case 2405300010005431

 

I've also worked around this by referencing the fully qualified path (not ideal but needed to move forward).

 

Thanks

Anonymous
Not applicable

Hi @bensimpsy ,

Thanks for sharing the support ticket.

Please allow some time, so team can check and provide a resolution.

In case if you got a resolution please do share with the community as it can be helpful to others .

Anonymous
Not applicable

Hi @bensimpsy ,

Thanks for using Fabric Community.
I assume you are taking about Deployment Pipelines.
Did you got a chance to look into this?  deployment-pipelines/create-rules 

vgchennamsft_0-1716546195467.png


If your is query is different, can you please explain with little more context so that I can guide you better.

Hope this is helpful. Please do let me know incase of further queries.

Helpful resources

Announcements
Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount!

November Carousel

Fabric Community Update - November 2024

Find out what's new and trending in the Fabric Community.

Dec Fabric Community Survey

We want your feedback!

Your insights matter. That’s why we created a quick survey to learn about your experience finding answers to technical questions.