Forum Discussion
Git Integration Best Practices
- 9 months ago
svenchio Thank you for your response. The only deviation I have is when it comes to defining the default lakehouse. We are going to have hundreds of notebooks, and defining a delpoyment rule for each one seems very tedious. What I am thinking of doing, is not connecting to any lakehouse via the UI. I would then use the %%config block and define my default lakehouse using a variable library.
Thanks again for your response. I've referenced some of your posts and videos before and they have been helpful.
Hi ABMN I KNOW EXACTLY what you're facing, so much so that I develope a custom tool, I called it FabricCatalyst in this youtube video I explain what it is Unlocking Fabric Automation: Lessons, Possibilities, and Real-World Insights ; this is just to show you that I investigated this subject quit extensively and I know what's happening behind the scenes, I had to build the custom deployment to achieve a fully funtional CI/CD process as Fabric.
But going with your issue, we need to split this in two parts cycles, one is the development cycle (cycle #1) and the other is the deployment cycle (cycle #2). The development cycle start from a workspace that serves as the integration where all feature branches merge their code to, this is the branch from where your 10 developers branch out to start a new feature, so, let's give this branch the name Integration and all the developers just sanboxes for simplicity.
The second cycle is the deployment, and once again, this integration workspace is the one that serves to start the deploymne to upper environments, usually, non-development like UAT and PROD, adn this environments in my professional opinion is BEST TO NOT HAVING ATTACHED TO BRANCHES! This is exacly what option #3 depicts 😁
The following assumes that you want for EACH DEVELOPER works on its OWN SANBOX LAKEHOUSE! I mean, if it would be allowed for the same Lakehouse in the integration workspace to be the central engine for all notebook, we would not be discussing this issue, right?
For the cycle #1, what need to happens is that you modify the notebooks' source code removing those "extra connection" on it's way back from the feature branch to it's origin, and for this, there's just TWO OPTION as of now, build a custom script that does that for you, or basically, "agree" with your developer team to remove their sandbox references as a condition to merge their feature branches back to integration. ... I think as of now, the human approach seems cheaper and faster, that the custom, your pick depends on the time and effort you would like to spend.
For the cycle #2, we are in a MUCH BETTER PLACE, because the pipelines ALLOW YOU TO CREATE deployment rules, and one of the rules currently supported is to define for each notebook what is the default lakehouse!!!
In othger words, via deployment rules you can define for each notebook what it's default lakehouse on each environment! This aricle describes what are and how to create deployment rules Create deployment rules for Fabric's ALM - Microsoft Fabric | Microsoft Learn
Well it's a long reply but it was worht explaining as I wanted to publish an article about this on my blog using your question, I think is a very interesting topic and I wanted to give it the time, hopefully you agree to kudos my response and mark as a solution. All the best, reach out to me if you want any additional information
- ABMN9 months ago
Helper I
svenchio Thank you for your response. The only deviation I have is when it comes to defining the default lakehouse. We are going to have hundreds of notebooks, and defining a delpoyment rule for each one seems very tedious. What I am thinking of doing, is not connecting to any lakehouse via the UI. I would then use the %%config block and define my default lakehouse using a variable library.
Thanks again for your response. I've referenced some of your posts and videos before and they have been helpful.