Forum Discussion
Deployment Pipeline for OneLake
hello.
I have a deployment pipeline setup as the usual Dev, Test and Production workspace. We use the medallion architeture, bronze, silver and gold lakehouses. I just want to clarify some of the similar answers to my question.
1. Each workspace should have its own pipeline
2. the deployment pipeline just copies over metadata for lakehouses and semantic models
3. currently there is no way for semantic models to bind to their target wokspace and lakehouse...targets will be pointed back to original.
So generally, for Onelake items the deployment pipeline tool is limited. We don't have an Azure Devops solution or plan to have one $$$$.
I'm trying to figure out how to organize this and what are best practices to leveraging the deployment pipeline. meaning, when working on the dev workspace and making changes to notebooks etc, it is assumed all three workspaces should already have the same lakehouse table data. just feel the deployment pipeline is redundant and i'm not using it to its full potential.
Hi grdelgado,
What storage mode are you using for your semantic models?
Import, DirectQuery, and DirectLake on SQL connections can all be remapped with deployment pipelines. DirectLake on OneLake cannot yet.If you're using DirectLake on OneLake, you can use the python ci-cd library to remap the connections, or you can do it manually in VS Code with Fabric Studio.
As far as lakehouses, it is intentional that the deployment pipelines do not copy over data. Imagine mixing your DEV data into PRD, that's a recipe for disaster.
I build all of my notebooks so that they are idempotent, meaning they will always produce the same output regardless of how or when they are run. All of my notebooks have a check to see if their target tables exist, and if not, they will create them. Once I deploy them to a new workspace (from dev to tst, or tst to prod), all I need to do is run them and all the tables that need to exist will be created during that first run.
4 Replies
Hi grdelgado,
What storage mode are you using for your semantic models?
Import, DirectQuery, and DirectLake on SQL connections can all be remapped with deployment pipelines. DirectLake on OneLake cannot yet.If you're using DirectLake on OneLake, you can use the python ci-cd library to remap the connections, or you can do it manually in VS Code with Fabric Studio.
As far as lakehouses, it is intentional that the deployment pipelines do not copy over data. Imagine mixing your DEV data into PRD, that's a recipe for disaster.
I build all of my notebooks so that they are idempotent, meaning they will always produce the same output regardless of how or when they are run. All of my notebooks have a check to see if their target tables exist, and if not, they will create them. Once I deploy them to a new workspace (from dev to tst, or tst to prod), all I need to do is run them and all the tables that need to exist will be created during that first run.
Thankyou @GilbertQ , @Lozovskyi , @tayloramy for Addressing the issue.
Hi @grdelgado ,
Thank you for reaching out to Microsoft Fabric Community Forum,
As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the superusers for the issue worked? or Let us know if you need any further assistance?
Regards,
Chaithanya
Thankyou GilbertQ , Lozovskyi , tayloramy for Addressing the issue.
Hi grdelgado ,
Thank you for reaching out to Microsoft Fabric Community Forum,
As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the superusers for the issue worked? or Let us know if you need any further assistance?
Regards,
Chaithanya
