Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
AdamFry
Advocate I
Advocate I

Deployment pipeline and lakehouse content

Hi there,

I'd like to understand the deployment pipeline functionality when it comes to working with lakehouses.  I have two development workspaces named bronze and silver.  Each of the workspaces has a lakehouse.  And the bronze lakehouse has some folders and files.  The silver lakehouse has a shortcut to the bronze lakehouse root folder.  I noticed that when I deploy the dev workspaces to my test workspaces, the lakehouses are created but they are empty.  Is this intentional?  I have other fabric items in my workspaces like notebooks and pipelines that depend on the content inside the lakehouse so I'm wondering how people handle this situation when deploying their workspace from dev to test to production?  Is there some certain amount of manual setup that must follow the deployment pipeline to configure structures and shortcuts in your lakehouse?  Do you have some method to generate those items?  I'm trying to understand the best practices around setting these things up.  Any help would be greatly appreciated!  

2 ACCEPTED SOLUTIONS
v-gchenna-msft
Community Support
Community Support

Hi @AdamFry ,

Thanks for using Fabric Community.

Right now, the Lake houses are created empty. We are working on improving that and getting the definition of the lakehouse to be exported and deployed. It will come in pieces so it will take time until the full definition will become available.


What we recommend to do with current options:

 

  • use data pipelines and notebooks to build the Lakehouse as much as possible. Since they are source controlled and deployed with their definition, you can manage the Lakehouse content and changes by deploying the data pipeline/notebook and running it to apply the changes into the Lakehouse.
  • For things that can't be managed through data pipeline/notebook, it will still need to happen manually. It needs to be configured once and then it will not be overridden in later deployments. For example- the shortcut to the bronze LH.


Docs to refer -
Lakehouse deployment pipelines and git integration - Microsoft Fabric | Microsoft Learn
Best practices for lifecycle management in Fabric - Microsoft Fabric | Microsoft Learn

Hope this is helpful. Please let me know incase of further queries.

View solution in original post

Hi Adam, Fabric deployment pipeline allows creating deployment rules for notebooks that parametrize default Lakehouse connection. More about this is here: https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-source-control-deployment

View solution in original post

6 REPLIES 6
Madalina2801
Frequent Visitor

Hi everyone,

 

I have another question related to this topic. How can we deploy data pipelines across workspaces? What is the most efficient way to achive this?

If I have a data pipeline with different activities (copy data, dataflows) that have different sources/destinations how can I change them during the deployment process? What are the best practices for deploying data pipelines?

 

Thanks in advance!

v-gchenna-msft
Community Support
Community Support

Hi @AdamFry ,

Thanks for using Fabric Community.

Right now, the Lake houses are created empty. We are working on improving that and getting the definition of the lakehouse to be exported and deployed. It will come in pieces so it will take time until the full definition will become available.


What we recommend to do with current options:

 

  • use data pipelines and notebooks to build the Lakehouse as much as possible. Since they are source controlled and deployed with their definition, you can manage the Lakehouse content and changes by deploying the data pipeline/notebook and running it to apply the changes into the Lakehouse.
  • For things that can't be managed through data pipeline/notebook, it will still need to happen manually. It needs to be configured once and then it will not be overridden in later deployments. For example- the shortcut to the bronze LH.


Docs to refer -
Lakehouse deployment pipelines and git integration - Microsoft Fabric | Microsoft Learn
Best practices for lifecycle management in Fabric - Microsoft Fabric | Microsoft Learn

Hope this is helpful. Please let me know incase of further queries.

I've marked this as a solution but I have another question, please let me know if it makes more sense for me to just create a new topic/thread since this is related but a bit different.  Again related to the release pipelines and lakehouses but now considering how release pipelines handle the lakehouse sources in notebooks.  

 

I have bronze_dev workspace with a lakehouse named lakehouse_bronze.  Now if I add a notebook in the bronze workspace which has the lakehouse_bronze added as a lakehouse source and my notebook reads from the files in the lakehouse_bronze and writes them as delta tables in lakehouse_bronze, when I publish this workspace from bronze_dev to bronze_test, am I correct that the notebook will be created in bronze_test but the lakehouse that the notebook references is still the lakehouse in bronze_dev?  Is it a normal practice after deployment to go through all the notebooks and reconfigure the lakehouse sources in them?  Is there a way for the deployment pipeline to automatically update that lakehouse reference?  

Hi Adam, Fabric deployment pipeline allows creating deployment rules for notebooks that parametrize default Lakehouse connection. More about this is here: https://learn.microsoft.com/en-us/fabric/data-engineering/notebook-source-control-deployment

Thank you so much, I totally missed the topic of deployment rules in the deployment pipelines. I really appreciate the help getting me in the right direction!

Thank you so much. That makes sense. For now I will look at creating a setup notebook and pipeline that can configure all the manual bits.  It would be amazing if the deployment pipeline had an option to run a specified arbitrary pipeline or notebook after the deployment process to handle these types of configurations. 

Helpful resources

Announcements
Europe Fabric Conference

Europe’s largest Microsoft Fabric Community Conference

Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.

Expanding the Synapse Forums

New forum boards available in Synapse

Ask questions in Data Engineering, Data Science, Data Warehouse and General Discussion.

RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

MayFBCUpdateCarousel

Fabric Monthly Update - May 2024

Check out the May 2024 Fabric update to learn about new features.