Getting tables from Warehouse to Lakehouse

[Ashwath_Bala_S]

Hi,

I have few queries regarding building Medallion Architecture in Fabric,

(i) Firstly, I would like to have Warehouse (Getting data from SQL to Warehouse using Pipeline) as a Bronze Layer and a Lakehouse (for Silver, Gold Layer).
Is there any approach to get data "from Warehouse using Spark Notebooks" and write the same in Lakehouse. Is there any approach to do that using Spark Notebooks in Lakehouse (I need to get bronze layer from Warheouse and have the silver and gold layer in Lakehouse, with the help of notebook)?

(ii) Secondly, If suppose, if I have Bronze, SIlver layers in Lakehouse, is there a way to get the gold layer in Warehouse (By converting silver tables into Gold in Notebooks, and then converting gold into Parquet format to save in "Files", and then getting the parquet format in Warehouse- to have my gold layer separately from Bronze and Silver).

Also, Can we automate the processes, such that tables can be updated based on incremental loads. (for both methods, if possible)

Any assistance on this would be highly helpful!

Any reference on the same will also be helpful!

Thanks in Advance!

[v-nikhilan-msft]

Hi @Ashwath_Bala_S 
Thanks for using Fabric Community.
Our engineering team is working on SQL Analytics connector to read data from Spark notebooks in Fabric. This feature is currently in currently in testing.  We do not have any ETA yet.

COPY INTO from Lakehouse files is currently planned. So if you want to load data from Lakehouse into Warehouse from files section of Lakehouse, it is possible. Again, this is in the roadmap. We do not have any ETA yet.

Hope this helps. Please let me know if you have any further queries.

[Ashwath_Bala_S]

Hi @v-nikhilan-msft ,

Thanks for your reply!

So, currently,

(i) Getting tables from Warehouse, using notebooks in Lakehouse and writing the same in Lakehouse (as delta tables as gold layer), and

(ii) Lakehouse Tables to files into Parquet format and into Warehouse as delta tables;

Both are in roadmap and testing phases, and we are waiting for ETA.

Kindly correct if my understanding is right!

Also, Is there any suggestion and the best alternative for hosting medallion architecture in Fabric.

Thanks in Advance!

[v-nikhilan-msft · Accepted solution]

Hi @Ashwath_Bala_S 
Yes you understood correct.
Ultimately, the best choice for hosting Medallion Architecture depends on your specific requirements, data volume, and desired level of control and flexibility. Please refer to these links to understand various scenarios:

https://medium.com/@mariusz_kujawski/exploring-the-medallion-architecture-in-microsoft-fabric-f19ad7...
https://medium.com/@valentin.loghin/medallion-architecture-in-microsoft-fabric-567d044c2ade
https://endjin.com/what-we-think/talks/microsoft-fabric-lakehouse-and-medallion-architecture
https://sqlserverbi.blog/2024/03/18/moving-from-power-bi-to-microsoft-fabric/

Hope this helps. Please let me know if you have any further questions.

[Ashwath_Bala_S]

Hi @v-nikhilan-msft ,

Yeah it helps!

Thank You!

[v-nikhilan-msft]

Hi @Ashwath_Bala_S 
Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.