Forum Discussion
Delta table maintenance - is it managed?
Hi frithjof_v
Yes, you’re correct in your understanding of the differences between Lakehouse and Data Warehouse in terms of time travel retention and vacuuming.
Lakehouse:
In Lakehouse, the vacuuming process is part of the Table Maintenance feature. This feature includes the Vacuum operation, which removes old files no longer referenced by a Delta table log. The files need to be older than the retention threshold, and the default file retention threshold is seven days.
Use table maintenance feature to manage delta tables in Fabric - Microsoft Fabric | Microsoft Learn
If you don’t run any maintenance, the Lakehouse tables will keep accumulating all the data until the Vacuum operation is executed. So, if you don’t choose to run vacuuming, then the Lakehouse tables will never be vacuumed.
Data Warehouse:
In Data Warehouse, the data retention for time travel queries is seven days. This is a limit across the OneLake for the entire Fabric. You can’t change it as of now. Therefore, Data Warehouse will be limited to 7 days time travel, and Fabric automatically performs the vacuuming without you being able to control it.
Time travel in Warehouse within Microsoft Fabric (preview) - Microsoft Fabric | Microsoft Learn
Hope this helps. Please let me know if you have any further questions.
Thank you Anonymous,
If I don't run any maintenance (more specifically: vacuuming) on the Lakehouse delta tables, will the tables keep accumulating all the data and not delete any data?
So if I don't choose to run vacuuming, then the Lakehouse tables will never be vacuumed?
Or does Fabric run vacuuming of the Lakehouse tables in the background by itself, even if I don't do anything to execute the vacuuming?
I'm trying to understand, regarding Lakehouse, if I have the full control over the retention of the data, or if Fabric will delete old versions even if I don't do anything.
For the data warehouse I think Fabric will automatically delete versions older than 7 days. I don't think I can control that.
Is it correct that there is a difference between Lakehouse and Warehouse when it comes to whether I can control the time travel retention?
Can Lakehouse have time travel with as long retention period as I want (possibly 5 years or even more if I want to), and it only vacuums the tables if and when I tell it to vacuum the tables? So if I don't do anything with respect to vacuuming, the Lakehouse tables will never get vacuumed, and I can time travel through the entire history of the table?
Whereas Data Warehouse will be limited to 7 days time travel, and Fabric automatically performs the vacuuming without me being able to control it?
Thank you 😀
Hi frithjof_v
Yes, you’re correct in your understanding of the differences between Lakehouse and Data Warehouse in terms of time travel retention and vacuuming.
Lakehouse:
In Lakehouse, the vacuuming process is part of the Table Maintenance feature. This feature includes the Vacuum operation, which removes old files no longer referenced by a Delta table log. The files need to be older than the retention threshold, and the default file retention threshold is seven days.
Use table maintenance feature to manage delta tables in Fabric - Microsoft Fabric | Microsoft Learn
If you don’t run any maintenance, the Lakehouse tables will keep accumulating all the data until the Vacuum operation is executed. So, if you don’t choose to run vacuuming, then the Lakehouse tables will never be vacuumed.
Data Warehouse:
In Data Warehouse, the data retention for time travel queries is seven days. This is a limit across the OneLake for the entire Fabric. You can’t change it as of now. Therefore, Data Warehouse will be limited to 7 days time travel, and Fabric automatically performs the vacuuming without you being able to control it.
Time travel in Warehouse within Microsoft Fabric (preview) - Microsoft Fabric | Microsoft Learn
Hope this helps. Please let me know if you have any further questions.
Thank you Anonymous,
I'm struggling to understand this information: "However, all the delta tables in OneLake have the same retention period."
Let's say I run vacuum on a Lakehouse table. Then I can choose the retention period for that vacuum operation freely. If I run a vacuum operation on another table, then I can choose another retention period for that vacuum operation.
So I would say that all tables don't have the same retention period. The retention period is chosen by me whenever I choose to run a vacuum operation on a single table. Or am I missing something?
Thank you 😀
Hi again frithjof_v
Yes you are right. I have edited the above post. The actual retention period for each table can be set individually based on your requirements when you perform a vacuum operation.
Hope this helps.😊