Forum Discussion
Bronze Layer Architecture
Dears,
Hope this message finds you well
I am designing a new fabric architecture in medallion format
I am in Bronze layer design
I have some different sources:
1) A Dataverse Direct Link to Fabric which creates a dataverse generated lakehouse, in my bronze layer, and adds some shortcuts into my OneLake. This Lakehouse does not allow me to choose the schemas where the links of the tables are added. Instead, those all fell under DBO schema.
2) I have as well a SQL Server Managed Instance Mirror crated from Fabric. This will copy (replicate) the information into a DWH generated to my bronze worskpace. This is a DWH auto-generated, its not something that I can tune, as far as I udneratand and its read only. As for the schemas inside the DWH, seems they are copied from the SQL Server MI, hence I cannot control the schemas where the tables will land. This is a read-only, near-realtime functionality
My questions:
I will probably have additional sources, like SQL Server on-premisses very soon
I will end up with different items in my bronze layer (lakehouse, DWH, etc..) and I have no control over the schemas for some of this objects. Based on that, I was thinking, still on the bronze layer, to create an additional lakehouse. But this one would have schemas like Source.Table1, Source.Table2, etc... So that I have the things structured in the bronze layer.
For that, I wqas thinking in creating some SQL Views or Shortcuts from this organized lakehouse into the other items (lakehouse and DHW).
This way, my Silver layer would only comunicate with this organized lakehouse
1) What do you think? would you do it as well? and what would you use to bring the objects from the other items into this lakehouse? Shortcuts or Views?
2) Can I use Views or shortcuts into this organzied lakehouse from both (lakehouse and DWH) or this is only possible between same object types like LH-to-lake house?
3) From a archiecture point of view, what shall I call this organized lakehouse? because of the the other times I can name them Raw data , but what about the organized lakehouse? what shall I call it?
Thanks a lot,
Pedro
If you want to know how I would do it then:
I would create Shortcuts in BRONZE layer from RAW to get what's required.
5 Replies
HI fabricpribeiro
I agree with the approach that you mentioned above regarding organised data. I have a suggestion here in case you want to keep your BRONZE layer clear.In your medallion architecture you can actually introduce a starting stage called RAW. Here you can bring all your data from multiple data sources without any transformations. This RAW layer can also have both Lakehouses & Warehouses. You can also have your RAW layer as a different workspace.
Then in the BRONZE layer you bring the data further to move to next stages like SILVER & GOLD. You can also organise your data in BRONZE layer.
When coming to using Shortcuts, it completly depends on what you are planning to do. But yes you can have shortcuts.
Reagrding naming your medallion architecture, I suggest:
RAW --> BRONZE --> SILVER --> GOLD
Just point here on number of layers in medallion architecture. It is completly up to your requirements on definining number of layers on the medallion architecture. Like you have a PLATINUM layer in some scenarions after GOLD or you have a PRE-BRONZE layer before BRONZE.
Thanks a lot for the answers. Appreciated
How about the data to be organized in the bronze lakehouse ?
The lakehouse which I will have schemas.
Can I link the raw data to it, via views ? and what about via shortcuts?
I mean, I will have for instance one lakehouse in raw and a DWH in raw. Can I use views in the bronze lakehouse to point to the data in the raw layer (for those two items) or would you use instead shortcuts?
This knowing that I will aim to have something like this in the bronze lakehouse :
source.Table1
Weather this is done via a view or via shortcut
Another question that I have is, we discussed about the RAW layer for data sources like Dataverse Fabric Direct Link and SQL Server Managed Instance Mirroring. This because we don't have control over schemas nor about the type of item (lakehouse or DHW) which the funcionality can create
But what about other sources, for example, a data pipeline to connect to SQL Server on-prem and take data? most probably, there I can while bringing the data incrementally and have control to state to which schema I wand to land it. Would you add also that into the raw? or that one you would already import it to the bronze?
Something like, if we have no control over schemas or items, then raw. Other wise, go directly t o Bronze? What is your opinion?
Thank you,
Pedro
I will bring all the raw data to my RAW layer whether it is views, via shortcuts or via pipelines. The reason being, I have all the raw data at one place which can be used by other teams to use further for their analysis.
BRONZE layer I will keep organised data, though still in raw form without any transformations.
Further SILVER layer can have all the transformations and business logics built in.
GOLD layer is final layer which has the transformed data which can be uses for semantic modelling.
