Fabric Medallion Architecture - Silver Lakehouse to Gold Warehouse

[VirajD]

Hi Everyone,

   We are in the process of designing a Medallion architecture in Fabric where our Bronze and Silver layers would be in Lakehouse and our Gold Layer in the Warehouse. 

What would be the best approach to load data from Silver Lakehouse to Gold Warehouse? 

[v-prasare]

Hi @VirajD,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

@R1k91,@Witmanm   Thanks for your prompt response

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

[Witmanm · Accepted solution]

The best approach would be one that takes your specific situation into account. It's not necessarily a "one size fits all" thing. This could be your teams' technical capabilities, the level of data transformations that need to be done, CU performance considerations, data scheduling, etc. 

If you're looking to do some simple transformations and your team is more comfortable with UI-based work flows, you can utilize Dataflow Gen 2s to handle your transition from the silver layer to gold. 

Data Pipelines would be useful if you want to orchestrate some more complicated logic. 

If your team is python savvy or you need to do some heavy transformations, consider notebooks. 

Each option has pros and cons regarding the considerations described earlier. 

You can refer to this decision guide in the Fabric documentation: https://learn.microsoft.com/en-us/fabric/fundamentals/decision-guide-pipeline-dataflow-spark 

[R1k91]

I would add my 2 cents, you should consider you could have different performances performing the same task on different workload.
for example Dataflows Gen2 automatically switch to Pipelines engine reaching a specific number of rows when "Fast copy" is enabled to speed up the copy process.