Forum Discussion
Post PatronBronze Layer Architecture
If you want to know how I would do it then:
I would create Shortcuts in BRONZE layer from RAW to get what's required.
Super UserHI fabricpribeiro
I agree with the approach that you mentioned above regarding organised data. I have a suggestion here in case you want to keep your BRONZE layer clear.
In your medallion architecture you can actually introduce a starting stage called RAW. Here you can bring all your data from multiple data sources without any transformations. This RAW layer can also have both Lakehouses & Warehouses. You can also have your RAW layer as a different workspace.
Then in the BRONZE layer you bring the data further to move to next stages like SILVER & GOLD. You can also organise your data in BRONZE layer.
When coming to using Shortcuts, it completly depends on what you are planning to do. But yes you can have shortcuts.
Reagrding naming your medallion architecture, I suggest:
RAW --> BRONZE --> SILVER --> GOLD
Just point here on number of layers in medallion architecture. It is completly up to your requirements on definining number of layers on the medallion architecture. Like you have a PLATINUM layer in some scenarions after GOLD or you have a PRE-BRONZE layer before BRONZE.
fabricpribeiroThanks a lot for the answers. Appreciated
How about the data to be organized in the bronze lakehouse ?
The lakehouse which I will have schemas.
Can I link the raw data to it, via views ? and what about via shortcuts?
I mean, I will have for instance one lakehouse in raw and a DWH in raw. Can I use views in the bronze lakehouse to point to the data in the raw layer (for those two items) or would you use instead shortcuts?
This knowing that I will aim to have something like this in the bronze lakehouse :
source.Table1
Weather this is done via a view or via shortcut
Another question that I have is, we discussed about the RAW layer for data sources like Dataverse Fabric Direct Link and SQL Server Managed Instance Mirroring. This because we don't have control over schemas nor about the type of item (lakehouse or DHW) which the funcionality can create
But what about other sources, for example, a data pipeline to connect to SQL Server on-prem and take data? most probably, there I can while bringing the data incrementally and have control to state to which schema I wand to land it. Would you add also that into the raw? or that one you would already import it to the bronze?
Something like, if we have no control over schemas or items, then raw. Other wise, go directly t o Bronze? What is your opinion?
Thank you,
Pedro
- Pragati11
I will bring all the raw data to my RAW layer whether it is views, via shortcuts or via pipelines. The reason being, I have all the raw data at one place which can be used by other teams to use further for their analysis.
BRONZE layer I will keep organised data, though still in raw form without any transformations.
Further SILVER layer can have all the transformations and business logics built in.
GOLD layer is final layer which has the transformed data which can be uses for semantic modelling.- fabricpribeiro
Thanks Pragati11 ,
But my question, in this particular case (of views Vs shorcuts) was more :
In the bronze layer, shall I create the schemas I wish and then connect to RAW via Shortcut or via Views? How would you do it?
Would it be more benefic shortcuts or Views ?
I will have two lakehouses and one DHW in Raw layer