Forum Discussion
How do I organize everything?
Dear HimanshuS-msft
Current challenges
We have been making reports and dashboards for several years now, connecting directly to our operational database(s). Of course, this is not a sustainable solution and we have had problems with this in the past.
We are currently making a plan to stop having to connect directly to our operational databases, typically that involves setting up some kind of storage solution in between.
With the release of Fabric, this is now relatively child's play. Instead of having to research the best combination of services in Azure, we can now have one portal with one capacity that can provide in all our BI & analytics needs.
Also, we are constantly improving our operations and have hired some people in the past on a project-basis to create analytical models for us. But we have never really been able to deploy them in our operations, because we lacked the know-how on how to deploy such models. Again, Fabric can help us with that by introducing notebooks. With notebooks, we can copy the code of the model, drop it in a notebook, and run it periodically on our data.
Regarding organizing + governance
My current view on things is that we will create at least 3 data lakehouses: bronze, silver & gold. But adding to that, I was wondering if I should make separate lakehouses for confidential data (such as HR and Finance) or whether I should manage access to that data within my B/S/G lakehouses.
Also, should I put everything in the same workspace, or should I split these? What about my ingestion dataflows (prod db = oracle) and notebooks? Same workspace as my lakehouses, or different ones?
I'm generally looking for some best practices on how to organize my workspaces and lakehouse/warehouse instances to stimulate fast progress once everything is set up, minimize rework when our data maturity has improved, but most of all keep everything crystal clear.
Thanks for your insights.
Hi @HimanshuS-msft ,
We have similar doubt of your,
We aim to deploy a medallion architecture with the usual 3-layer approach (bronze/silver/golden). But after that point many questions does apper all them related about how to organize the content
1) Lakehouses: 1 lakehouse by layer (Silver/..) or 1 lakehouse by layer/functional area (e.g: HR; finance;...)?
2) Fabric related artifacts (notebooks, pipelines, datadlows, lakehouses)
- 1 workspace by layer
- 1 workspace per layer/area and environment
- previous ones + sub-Folders (the long awaited future seems to arrive on 2024 Q1)
Has anyone a proposal about how to organize the Fabric content of a Medallion Architecture to avoid ending with tons of objects on a workspace?
Thx