Forum Discussion
Differences between Data Warehouse and Lakehouse
Hi,
I'm missing something about your answer, or you are revealing something extremely new, almost opening the pandora box.
In order to explain, I need to remind two main points:
- The concept broadly accepted about onelake is that it's a single storage for all the different tools we will use. It doesn't mind if it's a lakehouse or data warehouse, the data will be stored in delta.
- It goes to the level I watched a speaker mentioning in one official microsoft session "The choice depends only on the kind of tool you like to use, because both are stored in delta on onelake"
But on your answer you are revealing:
"The data warehouse is not as sensitive to the small files problem as the lakehouse because it uses a different storage mechanism. The data warehouse stores files in blobs, which are large containers that can hold multiple files."
This makes a huge difference, changes the base concept of onelake about having everything in a single place and format.
The fact the warehouse uses a different storage format is critical because we need to understand this format. Different formats leads to different results and this affects the architectural changes we should make. It's much more than the basic technical differences we already know about.
The Kusto database I already know from another answer here and other sources as well it uses a custom format and put a shortcut in the onelake to be acessible. But the warehouse keep the files in the onelake, we can see the files in the onelake explorer. If the files are kept in the onelake, it's not clear how in one moment or another it doesn't endup suffering the same problems related to this kind of storage - Delta.
Is there a deeper documentation about this format?
Am I misunderstanding something ?
Kind Regards,
Dennes
Hi DennesTorres ,
Apologies for the delay in reply from our side.
I have taken help from the internal team and got the information as below:
Firstly, the Data Warehouse files will be compacted shortly and they are in fact VOrdered (this was a recent change we introduced a few weeks ago). The reason you're observing no updates to the delta log when a warehouse table is updated is because we do not currently reflect updates and deletes in the delta log, but we will very soon. We do not support VACUUM but we will do our own garbage collection shortly as well. So, this concludes that Warehouse uses VORDER.
To understand better about the differences between Lakehouse and Warehouse please refer this document: Choosing between Lakehouse and Warehouse in Microsoft Fabric | LinkedIn
Hope this helps. Please let us know if you have any further queries.