Forum Discussion
Helper IGit Integration Best Practices
svenchio Thank you for your response. The only deviation I have is when it comes to defining the default lakehouse. We are going to have hundreds of notebooks, and defining a delpoyment rule for each one seems very tedious. What I am thinking of doing, is not connecting to any lakehouse via the UI. I would then use the %%config block and define my default lakehouse using a variable library.
Thanks again for your response. I've referenced some of your posts and videos before and they have been helpful.
Super UserABMNI agree that option 3 in the article you refer to is a good starting point for your workflow. With such a large team of 10 developers, code-first approach, and an enterprise environment with CI/CD and parallel development you need a very disciplined and well defined workflow for your team. You did not mention other environment where you may plan to deploy to like test, pre-production, production.
With all that in mind, I propose a list of rules that may be helpful in organizing your workspaces, Git flow, CI/CD, and deployments.
1. Each named integration environment such as Dev, Test, Prod, etc. should have its own dedicated workspace with no developers being able to make changes in.
2. Depending on your release strategy (from a single Git branch or individual environmental branches) you may decide how those named integration workspaces are connected to Git. Example, Dev workspace is connected to a dev branch in Git, etc.
3. Developers have their individual workspaces where they have admin privileges for their development tasks. Each developer's workspace is connected to a developer's working branch in Git. As development work progresses, developers re-connect their individual workspaces to their current working branches on demand.
4. The main integration branches (i.e. Dev) should have policies preventing direct pushes and commits, and only allowing a PR-based Git flow for integrating changes.
5. Because named integration workspaces connected to the main integration branches cannot be updated manually by team members, an automated external DevOps pipeline(s) must be used to sync the workspaces from Git after the PR is completed.
6. Shared resources that contain data such as lakehouse, a warehouse, a mirrored database, etc. may need to reside in separate workspaces that do not allow any direct changes from development team(s). In such a scenario, an isuue with the item IDs could be partially eliminated, but likely not completely. DevOps pipeline should be responsible for replacing items IDs where necessary.
7. It is also possible to implement a hybrid deployment approach between environments using built-in pipelines triggered by an external DevOps pipeline. Such an approach may be useful when Variable libraries and replacable parameters can be automatically utilized by Fabric deployment pipelines.
8. The major rule I use: Git flow should be the backbone of any and all deployment in Microsoft Fabric. If you have items in Git you can deploy them to any Fabric workspace using Azure DevOps and you have a way to modify the items' parameters to adapt to a specific environment.
I apologize for a late response, but hope it could be useful for other community members as well.