Forum Discussion
Git Integration Best Practices
I am working with a team of about 10 developers in Fabric. We want to set up individual/feature workspaces for each developer that would then be synched with a corresponding feature branch in Git. We make use of a lot of notebooks that have a default lakehouse defined. The plan would be for the developers to do their work in their feature branch and then merge it to the main branch. The main branch would then be synched to a separate workspace. What I am finding is that when you open a notebook in the workspace that is synched with the main branch, the default lakehouse is still pointing to the lakehouse in the feature workspace that it originated from. We also plan to branch out from this workspace into the individual/feature workspaces as new feature branches are needed. So we could potentially have notebooks in the main branch referencing 10 different feature worksapces. I know you can use %%configure in a notebook to set the default lakehouse and make use of variable libraries to control that, but each time you branch out from the main workspace there would need to be some configuration that would need to be done. I'm looking for the best practice on this and what methods others have used to get around this, with the least amount of overhead. Also, the main workspace would be the dev stage in the deployment pipeline. So we are looking at option 3 in this article -> CI/CD workflow options in Fabric - Microsoft Fabric | Microsoft Learn
Thanks in advance for any help.
svenchio Thank you for your response. The only deviation I have is when it comes to defining the default lakehouse. We are going to have hundreds of notebooks, and defining a delpoyment rule for each one seems very tedious. What I am thinking of doing, is not connecting to any lakehouse via the UI. I would then use the %%config block and define my default lakehouse using a variable library.
Thanks again for your response. I've referenced some of your posts and videos before and they have been helpful.
10 Replies
Hi,
Since you have a decent number of developers involved as well it is worth looking into implementing the custom "Branch out to new workspace" offering by Microsoft that is available in GitHub. Which offers to do the rebinding automatically:
https://github.com/microsoft/fabric-toolbox/tree/main/accelerators/CICD/Branch-out-to-new-workspace
I hope this helps. If it does please give this solution kudos or accept it as the solution.KevinChant My apologies. I didn't take a look at your link before I sent my response. I thought you were referring to the out of the box branch out functionality. I will take a look at your link. Thanks.
That is fine, I hope it answers your original question for you. If so, please mark it as a solution.
Hi ABMN I KNOW EXACTLY what you're facing, so much so that I develope a custom tool, I called it FabricCatalyst in this youtube video I explain what it is Unlocking Fabric Automation: Lessons, Possibilities, and Real-World Insights ; this is just to show you that I investigated this subject quit extensively and I know what's happening behind the scenes, I had to build the custom deployment to achieve a fully funtional CI/CD process as Fabric.
But going with your issue, we need to split this in two parts cycles, one is the development cycle (cycle #1) and the other is the deployment cycle (cycle #2). The development cycle start from a workspace that serves as the integration where all feature branches merge their code to, this is the branch from where your 10 developers branch out to start a new feature, so, let's give this branch the name Integration and all the developers just sanboxes for simplicity.
The second cycle is the deployment, and once again, this integration workspace is the one that serves to start the deploymne to upper environments, usually, non-development like UAT and PROD, adn this environments in my professional opinion is BEST TO NOT HAVING ATTACHED TO BRANCHES! This is exacly what option #3 depicts 😁
The following assumes that you want for EACH DEVELOPER works on its OWN SANBOX LAKEHOUSE! I mean, if it would be allowed for the same Lakehouse in the integration workspace to be the central engine for all notebook, we would not be discussing this issue, right?
For the cycle #1, what need to happens is that you modify the notebooks' source code removing those "extra connection" on it's way back from the feature branch to it's origin, and for this, there's just TWO OPTION as of now, build a custom script that does that for you, or basically, "agree" with your developer team to remove their sandbox references as a condition to merge their feature branches back to integration. ... I think as of now, the human approach seems cheaper and faster, that the custom, your pick depends on the time and effort you would like to spend.
For the cycle #2, we are in a MUCH BETTER PLACE, because the pipelines ALLOW YOU TO CREATE deployment rules, and one of the rules currently supported is to define for each notebook what is the default lakehouse!!!
In othger words, via deployment rules you can define for each notebook what it's default lakehouse on each environment! This aricle describes what are and how to create deployment rules Create deployment rules for Fabric's ALM - Microsoft Fabric | Microsoft Learn
Well it's a long reply but it was worht explaining as I wanted to publish an article about this on my blog using your question, I think is a very interesting topic and I wanted to give it the time, hopefully you agree to kudos my response and mark as a solution. All the best, reach out to me if you want any additional information
svenchio Thank you for your response. The only deviation I have is when it comes to defining the default lakehouse. We are going to have hundreds of notebooks, and defining a delpoyment rule for each one seems very tedious. What I am thinking of doing, is not connecting to any lakehouse via the UI. I would then use the %%config block and define my default lakehouse using a variable library.
Thanks again for your response. I've referenced some of your posts and videos before and they have been helpful.
Hi ABMN,
The ID of the default lakehouse is defined in the META tags in lines 10-14 of the notebook-content.py file that is synced in Git.
You can set up GitHub Actions or DevOps Pipelines to modify this ID when merging to main.
You also can use deployment pipeline rules to change this I believe.
Not an ideal solution, but workable.
If you found this helpful, cosnider giving Kudos. If I solved your problem or answered your question, mark this post as a solution.
Hi ABMN ,
I think one approach is to use Variable library and %%configure as you suggested with few adjusment.
you main workspace which connected to main branch will use the default value set. (main)
each time a developer is branching out, the developer will need to add a new value-set adding the correct id and set this value-set as active for this workspace.
When the PR was made from feature branch to main branch - use a fabric-cicd tool https://microsoft.github.io/fabric-cicd/0.1.30/ to deploy to main workspace, this will ensure that main workspace notebooks keep pointing to same lakehouse in main and not to feature workspace.
once the deployment using the fabric-cicd tool is complete, call the commit-to-git api, https://learn.microsoft.com/en-us/rest/api/fabric/core/git/commit-to-git?tabs=HTTP to commit the changes into the main branch.
ABMNI agree that option 3 in the article you refer to is a good starting point for your workflow. With such a large team of 10 developers, code-first approach, and an enterprise environment with CI/CD and parallel development you need a very disciplined and well defined workflow for your team. You did not mention other environment where you may plan to deploy to like test, pre-production, production.
With all that in mind, I propose a list of rules that may be helpful in organizing your workspaces, Git flow, CI/CD, and deployments.
1. Each named integration environment such as Dev, Test, Prod, etc. should have its own dedicated workspace with no developers being able to make changes in.
2. Depending on your release strategy (from a single Git branch or individual environmental branches) you may decide how those named integration workspaces are connected to Git. Example, Dev workspace is connected to a dev branch in Git, etc.
3. Developers have their individual workspaces where they have admin privileges for their development tasks. Each developer's workspace is connected to a developer's working branch in Git. As development work progresses, developers re-connect their individual workspaces to their current working branches on demand.
4. The main integration branches (i.e. Dev) should have policies preventing direct pushes and commits, and only allowing a PR-based Git flow for integrating changes.
5. Because named integration workspaces connected to the main integration branches cannot be updated manually by team members, an automated external DevOps pipeline(s) must be used to sync the workspaces from Git after the PR is completed.
6. Shared resources that contain data such as lakehouse, a warehouse, a mirrored database, etc. may need to reside in separate workspaces that do not allow any direct changes from development team(s). In such a scenario, an isuue with the item IDs could be partially eliminated, but likely not completely. DevOps pipeline should be responsible for replacing items IDs where necessary.
7. It is also possible to implement a hybrid deployment approach between environments using built-in pipelines triggered by an external DevOps pipeline. Such an approach may be useful when Variable libraries and replacable parameters can be automatically utilized by Fabric deployment pipelines.
8. The major rule I use: Git flow should be the backbone of any and all deployment in Microsoft Fabric. If you have items in Git you can deploy them to any Fabric workspace using Azure DevOps and you have a way to modify the items' parameters to adapt to a specific environment.
I apologize for a late response, but hope it could be useful for other community members as well.
