Forum Discussion
Differences between Data Warehouse and Lakehouse
Hi DennesTorres ,
Thanks for using Fabric Community.
The differences between the lakehouse and the data warehouse are due to the fact that they are different types of data storage systems with different design goals and trade-offs.
The lakehouse is designed to provide a unified data platform that combines the benefits of data lakes and data warehouses. It uses a combination of Delta Lake and Azure Synapse Link to provide fast and efficient querying of large datasets, while also supporting real-time data ingestion and processing.
The data warehouse, on the other hand, is designed to provide a scalable and highly available data warehousing solution that can handle large volumes of structured data. It uses a columnar storage format and a massively parallel processing architecture to provide fast query performance and high scalability.
1) How the differences are positive for each one, lakehouse and data warehouse, although they are the opposite? For example, why the warehouse is not affected by small files (or it is?) ? Why the warehouse doesn't need the VORDER ?
The data warehouse is not as sensitive to the small files problem as the lakehouse because it uses a different storage mechanism. The data warehouse stores files in blobs, which are large containers that can hold multiple files. This means that the number of files in the data warehouse does not have as big of an impact on performance as it does in the lakehouse.
You cannot use VORDER for tables in Warehouse. VORDER is a file format that is optimized for columnar storage, but it is not supported by Warehouse.Warehouse uses a different storage mechanism for structured data, which is optimized for performance for complex queries on structured data.
2) After an update in the data warehouse table, new parquet files were created, as expected. But the delta_log files were not updated. Why?
Regarding this issue there is already an ongoing issue in Fabric and there is a bug created for the same.
You can check the issues in this link:
https://support.fabric.microsoft.com/en-IN/known-issues//
3) After creating a shortcut from the Data Warehouse to a lakehouse, the VACUUM didn't work. I imagine it doesn't because the new parquet files were not updated in the delta_log, am I correct? How to fix this?
Yes you are correct. As I already mentioned there is an ongoing bug regarding delta_log, the VACUUM command may not work because the new parquet files were not updated in the delta_log.
Conclusion
The lakehouse and data warehouse formats have different strengths and weaknesses. The lakehouse format is designed for a wide range of workloads, including large data queries, incremental updates, and machine learning. The data warehouse format is designed for workloads that require high performance
Hope this helps. Please do let us know if you have any further queries.
Hi DennesTorres ,
We haven’t heard from you on the last response and was just checking back to see if your query has been resolved. Please let us know if you have any further queries.
- Anonymous2 years agoNot applicable
Hi DennesTorres ,
Following up to check if your query has been resolved. Please do let us know if you have any further queries.