Supplies are limited. Contact info@espc.tech right away to save your spot before the conference sells out.
Get your discountScore big with last-minute savings on the final tickets to FabCon Vienna. Secure your discount
Hello all, I hope you're doing well!
This is my first post here, so please be gentle 😊 I'm a junior data engineer/scientist working at a small-to-midsized company, and I'm currently leading the creation of our data platform entirely on my own.
As you’ll see from the diagram, I’ve split responsibilities between Microsoft Fabric (for data engineering) and Power BI (for reporting and semantic modeling). I’d greatly appreciate feedback from experienced professionals on whether my architecture reflects common best practices, or if I may be steering toward complications later on.
Data platform architecture
Here the key points:
My Questions
I’d love your insights on a few points I’m still unsure about:
Thanks in advance for any feedback — I really want to make sure I'm building something solid and maintainable before our data needs scale!
Solved! Go to Solution.
Hi @ndamoi ,
Thank you for reaching out to the Microsoft fabric community forum.
To answer your first question: yes, keeping the bronze and silver layers centralized is a common and smart setup, especially for small or mid-sized companies. It helps you keep your data consistent, makes things easier to manage, and avoids repeating the same transformation logic in different places which is great when different teams use the same cleaned data.
Your approach of using department-specific workspaces with their own Gold layer and connecting them to shared Silver tables using shortcuts is a solid and flexible solution. It gives each team the freedom to add their own business logic without changing the core data. This setup is actually supported in Microsoft Fabric through Lakehouse Shortcuts, which are designed exactly for sharing data between workspaces without copying it.
Your separation of environments into Dev, Test, and Prod, along with connecting them to Git branches, demonstrates good planning for long-term stability. While full Git integration in Fabric is still being developed, using repositories and deployment pipelines as a temporary solution is effective and will position you well as more automation features become available.
Your approach to access control restricting Bronze and Silver to yourself while granting departments access to Gold lakehouses and Power BI workspaces follows current data ownership practices, balancing centralized engineering with decentralized consumption. This setup supports both data quality and governance, while allowing for secure self-service analytics. The main areas to watch are the complexity from shortcuts, metadata documentation, and auditing pipeline runs. It would help to clearly document shortcut usage and use consistent naming to make dependencies easier to follow. As your platform grows, consider adding simple operational logs or dashboards to monitor data freshness and ownership.
Overall, your setup follows good design practices and matches how Microsoft Fabric is meant to be used. It puts you in a great position to grow and manage things smoothly as your data needs increase.
Hope this helps. Please feel free to reach out for further assistance.
Thank you.
Hi @ndamoi ,
Thank you for reaching out to the Microsoft fabric community forum.
To answer your first question: yes, keeping the bronze and silver layers centralized is a common and smart setup, especially for small or mid-sized companies. It helps you keep your data consistent, makes things easier to manage, and avoids repeating the same transformation logic in different places which is great when different teams use the same cleaned data.
Your approach of using department-specific workspaces with their own Gold layer and connecting them to shared Silver tables using shortcuts is a solid and flexible solution. It gives each team the freedom to add their own business logic without changing the core data. This setup is actually supported in Microsoft Fabric through Lakehouse Shortcuts, which are designed exactly for sharing data between workspaces without copying it.
Your separation of environments into Dev, Test, and Prod, along with connecting them to Git branches, demonstrates good planning for long-term stability. While full Git integration in Fabric is still being developed, using repositories and deployment pipelines as a temporary solution is effective and will position you well as more automation features become available.
Your approach to access control restricting Bronze and Silver to yourself while granting departments access to Gold lakehouses and Power BI workspaces follows current data ownership practices, balancing centralized engineering with decentralized consumption. This setup supports both data quality and governance, while allowing for secure self-service analytics. The main areas to watch are the complexity from shortcuts, metadata documentation, and auditing pipeline runs. It would help to clearly document shortcut usage and use consistent naming to make dependencies easier to follow. As your platform grows, consider adding simple operational logs or dashboards to monitor data freshness and ownership.
Overall, your setup follows good design practices and matches how Microsoft Fabric is meant to be used. It puts you in a great position to grow and manage things smoothly as your data needs increase.
Hope this helps. Please feel free to reach out for further assistance.
Thank you.
Thanks so much for your quick and thoughtful response.
Your feedback is really reassuring and helps me feel more confident about the direction I’m taking. I’ll work on implementing the missing concepts and focus on documenting everything properly.
The tips you shared are super helpful and much appreciated.
Thanks again for your support!
User | Count |
---|---|
5 | |
4 | |
3 | |
3 | |
2 |
User | Count |
---|---|
10 | |
8 | |
7 | |
6 | |
6 |