Forum Discussion
Best Practice for Lakehouse Strategy
Hi bw_chec,
My preferred approach in these cases is to have one DEV/TEST/PROD deployment pipeline with one workspace per stage. In one workspace, you have all the required lakehouses (in your case bronze, silver and gold) and move them to the next stage with the deployment pipelines. All workspaces within the deployment pipeline contain all data. After you have your data ready in the production workspace, you create new workspaces per department. Within the department workspace, you create one lakehouse and make shortcuts to the production gold lakehouse tables (as you requested, silver and bronze are also possible) that department should have access to. The lakehouse you create for that department has a default semantic model attached that contains all the requested data for that department.
In that case, the data ETL flow is clear and managed in one central place. The different department workspaces only contain shortcuts, without modification of the data. This ensures integrity of the data with the ability to controll access to specific tables and files.
You can find the documentation of shortcuts here: https://learn.microsoft.com/en-us/fabric/onelake/onelake-shortcuts#lakehouse.Best regards,
Fabian
Hi bw_chec,
My preferred approach in these cases is to have one DEV/TEST/PROD deployment pipeline with one workspace per stage. In one workspace, you have all the required lakehouses (in your case bronze, silver and gold) and move them to the next stage with the deployment pipelines. All workspaces within the deployment pipeline contain all data. After you have your data ready in the production workspace, you create new workspaces per department. Within the department workspace, you create one lakehouse and make shortcuts to the production gold lakehouse tables (as you requested, silver and bronze are also possible) that department should have access to. The lakehouse you create for that department has a default semantic model attached that contains all the requested data for that department.
In that case, the data ETL flow is clear and managed in one central place. The different department workspaces only contain shortcuts, without modification of the data. This ensures integrity of the data with the ability to controll access to specific tables and files.
You can find the documentation of shortcuts here: https://learn.microsoft.com/en-us/fabric/onelake/onelake-shortcuts#lakehouse.
Best regards,
Fabian
Fabian you are a genius. This is by far the best suggestion for workspace / lakehouse structuring I have heard. So in your design, each department would have their own lakehouse which only contains a collection of shortcuts, therefore no further copies of data?
Hi nealamin,
Yes that is correct. One lakheouse per department, no copies, no adjustments, just a selection of required tables.