Forum Discussion
How do I organize everything?
Hi Noeleke1301
Thanks for you using Fabric community forum .
I am aware that many of us are excited about Fabric and want to transition to this . But do you have any current challenges which you have in the current setup which you are willing to resolve ?
I will suggest that you should go with #2
2) Or should I make several data lakehouses (one for each data 'category')?
This wil give you they option to set permission at the Lakehouse level one for each department .
Thanks
Himanshu
- Noeleke13013 years ago
Helper III
Dear HimanshuS-msft
Current challenges
We have been making reports and dashboards for several years now, connecting directly to our operational database(s). Of course, this is not a sustainable solution and we have had problems with this in the past.
We are currently making a plan to stop having to connect directly to our operational databases, typically that involves setting up some kind of storage solution in between.
With the release of Fabric, this is now relatively child's play. Instead of having to research the best combination of services in Azure, we can now have one portal with one capacity that can provide in all our BI & analytics needs.
Also, we are constantly improving our operations and have hired some people in the past on a project-basis to create analytical models for us. But we have never really been able to deploy them in our operations, because we lacked the know-how on how to deploy such models. Again, Fabric can help us with that by introducing notebooks. With notebooks, we can copy the code of the model, drop it in a notebook, and run it periodically on our data.
Regarding organizing + governance
My current view on things is that we will create at least 3 data lakehouses: bronze, silver & gold. But adding to that, I was wondering if I should make separate lakehouses for confidential data (such as HR and Finance) or whether I should manage access to that data within my B/S/G lakehouses.
Also, should I put everything in the same workspace, or should I split these? What about my ingestion dataflows (prod db = oracle) and notebooks? Same workspace as my lakehouses, or different ones?
I'm generally looking for some best practices on how to organize my workspaces and lakehouse/warehouse instances to stimulate fast progress once everything is set up, minimize rework when our data maturity has improved, but most of all keep everything crystal clear.
Thanks for your insights.
- alfBI2 years ago
Responsive Resident
Hi @HimanshuS-msft ,
We have similar doubt of your,
We aim to deploy a medallion architecture with the usual 3-layer approach (bronze/silver/golden). But after that point many questions does apper all them related about how to organize the content1) Lakehouses: 1 lakehouse by layer (Silver/..) or 1 lakehouse by layer/functional area (e.g: HR; finance;...)?
2) Fabric related artifacts (notebooks, pipelines, datadlows, lakehouses)
- 1 workspace by layer
- 1 workspace per layer/area and environment- previous ones + sub-Folders (the long awaited future seems to arrive on 2024 Q1)
Has anyone a proposal about how to organize the Fabric content of a Medallion Architecture to avoid ending with tons of objects on a workspace?
Thx