Forum Discussion
Architecture Design in Fabric
I am new to Data Engineering and Fabric, I am trying to design an architecture for my organization. My goal is to have a centralized data repository that allows high level of data governance, but not to complicated to implement, and easy to manage. After doing some research I have come up with two designs that I think would work well (Open to better solutions). I believe I need atleast a Dev/Prod/Dept workspaces to allow for testing, and separation of live production data that will be used by the org. Which solution is better, should I have lakehouses for the stages (Bronze, Silver, Gold) or by Dept (IT, HR, Finance, etc.) or does it not really matter? Users within the org who will be viewing the data will only be granted access to data within the Dept workspace which will be Mirrors/Shortcuts to the Gold Level data to the respective tables in the Produciton Workspace they need access to. To my understanding I will need to use Deployment Pipelines to transfer the data across Workspaces as they move across their stages (Bronze, Silver, Gold)
Hi JDay98 ,
There are few points based on my understanding, Decoupled architecture helps in maintaining data access to restricted set of people.
1. You can leverage Domain based in case if you want to separate it out at higher level by departments
2. Lakehouse Schema is still in Preview , so some features will not function as expected. Better to avoid it.
3. Have you explored One Lake Security?
4. With Fabric Deployment Pipeline you cannot copy data.
Regards,
Srisakthi
5 Replies
Hi JDay98 ,
There are few points based on my understanding, Decoupled architecture helps in maintaining data access to restricted set of people.
1. You can leverage Domain based in case if you want to separate it out at higher level by departments
2. Lakehouse Schema is still in Preview , so some features will not function as expected. Better to avoid it.
3. Have you explored One Lake Security?
4. With Fabric Deployment Pipeline you cannot copy data.
Regards,
Srisakthi
May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.
If we don’t hear back, we’ll go ahead and close this thread. For any further discussions or questions, please start a new thread in the Microsoft Fabric Community Forum we’ll be happy to assist.
Thank you for being part of the Microsoft Fabric Community.Use Option 2 (Department-wise Lakehouse) for ease of access control, lower overhead, and modular ownership.
Limit the access in Department Workspaces to only relevant Gold tables via Shortcuts (good for governance).
Use Data Pipelines or Notebooks to move data across stages (Bronze > Silver > Gold) within the same lakehouse.
Use Deployment Pipelines to promote your entire workspace (including lakehouse artifacts and PBI reports) from Dev to Prod.
