Hi,
My team is trying to work with branched workspaces, backed up by the git integration.
We have our development workspace, which is connected to the develop branch. Then, each developer branches off into a new workspace to develop privately in their own branch and workspace.
What we've found that works up to now:
- datawarehouse, sql db, notebooks, datapipelines are all backed up by git
- you can merge new tables, views etc in the warehouse from your branched workspace.
- we have found that when we build pipelines and notebooks using existing tables, creating new table or using Lakehouse data, we need to reference the lakehouses in the development workspace, and because its not backed up by git, when we merge, there is no errors or issues.
What we are finding challenging & need help with is in our development workspace, we have a lakehouse and datawarehouse. We would like to create a View or a Stored Procedure that references a Lakehouse table. When a workspace is branched, the lakehouse is empty (as described in the documentation), so to get around this, we created some shortcuts to the development workspace Lakehouse table from a branched workspace, so that a view could be created in the branched workspace datawarehouse and then merged into the development workspace/ develop branch. First off, we got errors due to shortcuts having the same names as existing development Lakehouse tables. Therefore, we removed the shortcuts from the develop branch and decided not to commit shortcuts from branched workspaces, which could easily be done. However, we found that when new developers pulled code or branched off to a new workspace, with a view referencing a Lakehouse table, this caused errors in the new branched workspace as there was no shortcut or Lakehouse table to reference, and was just not workable.
So, I next resorted to removing the datawarehouse from the git versioning by adding the datawarehouse's repo path to the .gitignore file e.g. 'datawarehouse.Warehouse/'. This works when I have the repo open locally in VS code, however, in the Fabric UI when something is changed in the datawarehouse, we still get a Source Control Changes alert icon appear, with the option to commit. The develop branch is protected, as we'd like to use the PR process to merge code.
So it seems that the .gitignore file is not being recognised either, which is frustrating. Is this right? Or am I doing something wrong? If this option worked then we were just going to control the datawarehouse deployments with its own db project deployment.
I know in the forum and documentation, people recommend creating lakehouse tables in their branched workspaces to be referenced, however, this just does not seem feasible when you have 1000s of lakehouse tables to reference.
Any recommendations or guidance on this would be most helpful.
Thanks!
