Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Grow your Fabric skills and prepare for the DP-600 certification exam by completing the latest Microsoft Fabric challenge.

Reply
Ashwath_Bala_S
Helper II
Helper II

Getting tables from Warehouse to Lakehouse

Hi,

 

I have few queries regarding building Medallion Architecture in Fabric,

 

(i) Firstly, I would like to have Warehouse (Getting data from SQL to Warehouse using Pipeline) as a Bronze Layer and a Lakehouse (for Silver, Gold Layer).
Is there any approach to get data "from Warehouse using Spark Notebooks" and write the same in Lakehouse. Is there any approach to do that using Spark Notebooks in Lakehouse (I need to get bronze layer from Warheouse and have the silver and gold layer in Lakehouse, with the help of notebook)?

 

(ii) Secondly, If suppose, if I have Bronze, SIlver layers in Lakehouse, is there a way to get the gold layer in Warehouse (By converting silver tables into Gold in Notebooks, and then converting gold into Parquet format to save in "Files", and then getting the parquet format in Warehouse- to have my gold layer separately from Bronze and Silver).

 

Also, Can we automate the processes, such that tables can be updated based on incremental loads. (for both methods, if possible)

 

Any assistance on this would be highly helpful!

Any reference on the same will also be helpful!

Thanks in Advance!

1 ACCEPTED SOLUTION

Hi @Ashwath_Bala_S 
Yes you understood correct.
Ultimately, the best choice for hosting Medallion Architecture depends on your specific requirements, data volume, and desired level of control and flexibility. Please refer to these links to understand various scenarios:

https://medium.com/@mariusz_kujawski/exploring-the-medallion-architecture-in-microsoft-fabric-f19ad7...
https://medium.com/@valentin.loghin/medallion-architecture-in-microsoft-fabric-567d044c2ade
https://endjin.com/what-we-think/talks/microsoft-fabric-lakehouse-and-medallion-architecture
https://sqlserverbi.blog/2024/03/18/moving-from-power-bi-to-microsoft-fabric/

Hope this helps. Please let me know if you have any further questions.

View solution in original post

5 REPLIES 5
v-nikhilan-msft
Community Support
Community Support

Hi @Ashwath_Bala_S 
Thanks for using Fabric Community.
Our engineering team is working on SQL Analytics connector to read data from Spark notebooks in Fabric. This feature is currently in currently in testing.  We do not have any ETA yet.

COPY INTO from Lakehouse files is currently planned. So if you want to load data from Lakehouse into Warehouse from files section of Lakehouse, it is possible. Again, this is in the roadmap. We do not have any ETA yet.

 

Hope this helps. Please let me know if you have any further queries.

Hi @v-nikhilan-msft ,

 

Thanks for your reply!

So, currently,

 

(i) Getting tables from Warehouse, using notebooks in Lakehouse and writing the same in Lakehouse (as delta tables as gold layer), and

(ii) Lakehouse Tables to files into Parquet format and into Warehouse as delta tables;

Both are in roadmap and testing phases, and we are waiting for ETA.

 

Kindly correct if my understanding is right!

 

Also, Is there any suggestion and the best alternative for hosting medallion architecture in Fabric.

 

Thanks in Advance!

Hi @Ashwath_Bala_S 
Yes you understood correct.
Ultimately, the best choice for hosting Medallion Architecture depends on your specific requirements, data volume, and desired level of control and flexibility. Please refer to these links to understand various scenarios:

https://medium.com/@mariusz_kujawski/exploring-the-medallion-architecture-in-microsoft-fabric-f19ad7...
https://medium.com/@valentin.loghin/medallion-architecture-in-microsoft-fabric-567d044c2ade
https://endjin.com/what-we-think/talks/microsoft-fabric-lakehouse-and-medallion-architecture
https://sqlserverbi.blog/2024/03/18/moving-from-power-bi-to-microsoft-fabric/

Hope this helps. Please let me know if you have any further questions.

Hi @v-nikhilan-msft ,

 

Yeah it helps!

Thank You!

Hi @Ashwath_Bala_S 
Glad that your query got resolved. Please continue using Fabric Community for any help regarding your queries.

Helpful resources

Announcements
RTI Forums Carousel3

New forum boards available in Real-Time Intelligence.

Ask questions in Eventhouse and KQL, Eventstream, and Reflex.

Expanding the Synapse Forums

New forum boards available in Synapse

Ask questions in Data Engineering, Data Science, Data Warehouse and General Discussion.

MayFabricCarousel

Fabric Monthly Update - May 2024

Check out the May 2024 Fabric update to learn about new features.

Top Kudoed Authors