PySpark Update Warehouse

Hi holgergubbels, as deborshi_nag and tayloramy mentioned above the confusion in this situation comes from misunderstanding the Fabric resources and how to use them. Here is a decision guide that may help you make the right choice between Warehouse and Lakehouse Microsoft Fabric Decision Guide: Choose between Warehouse and Lakehouse - Microsoft Fabric | Microsoft Learn.

My preference, when implementing a medallion architecture in Fabric, is to use a Lakehouse for both data landing, raw (bronze), and curated (silver) medallion tiers. I would use a Warehouse for consumption (gold) tiers only because dimensional modelling is better implemented in a Warehouse and consumption is usually better aligned with using SQL. From this point of view, Fabric tools map naturally: Fabric pipelines can orchestrate all batch operations, including but not limited to data ingestion, landing, validations and transformations. Complex data transformations are implemented with Notebooks, and data loading into a Warehouse using script activity or a stored procedure activity. The metadata could be stored in a Lakehouse (files or detla tables) or in a SQL database. I would advise against using a Warehouse as a metadata storage.

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.