Forum Discussion
Best Practice for Lakehouse Strategy
Hi bw_chec,
My preferred approach in these cases is to have one DEV/TEST/PROD deployment pipeline with one workspace per stage. In one workspace, you have all the required lakehouses (in your case bronze, silver and gold) and move them to the next stage with the deployment pipelines. All workspaces within the deployment pipeline contain all data. After you have your data ready in the production workspace, you create new workspaces per department. Within the department workspace, you create one lakehouse and make shortcuts to the production gold lakehouse tables (as you requested, silver and bronze are also possible) that department should have access to. The lakehouse you create for that department has a default semantic model attached that contains all the requested data for that department.
In that case, the data ETL flow is clear and managed in one central place. The different department workspaces only contain shortcuts, without modification of the data. This ensures integrity of the data with the ability to controll access to specific tables and files.
You can find the documentation of shortcuts here: https://learn.microsoft.com/en-us/fabric/onelake/onelake-shortcuts#lakehouse.Best regards,
Fabian
Hi bw_chec,
My preferred approach in these cases is to have one DEV/TEST/PROD deployment pipeline with one workspace per stage. In one workspace, you have all the required lakehouses (in your case bronze, silver and gold) and move them to the next stage with the deployment pipelines. All workspaces within the deployment pipeline contain all data. After you have your data ready in the production workspace, you create new workspaces per department. Within the department workspace, you create one lakehouse and make shortcuts to the production gold lakehouse tables (as you requested, silver and bronze are also possible) that department should have access to. The lakehouse you create for that department has a default semantic model attached that contains all the requested data for that department.
In that case, the data ETL flow is clear and managed in one central place. The different department workspaces only contain shortcuts, without modification of the data. This ensures integrity of the data with the ability to controll access to specific tables and files.
You can find the documentation of shortcuts here: https://learn.microsoft.com/en-us/fabric/onelake/onelake-shortcuts#lakehouse.
Best regards,
Fabian
Hi Fabian,
I know I'm late for the train for quite a bit but I'm curious about the "all workspaces within the deployment pipeline contain all data" part. How do you do that? The deployment pipeline only updates metadata of, for example, lakehouses (e.g. tables and columns within them) but not the data.
Where I'm stuck at is figuring out how to test my dev stuff on test data (in test stage and test workspace) when they're not in the same workspace. Thanks in advance!