Delta table maintenance - is it managed?

[frithjof_v]

Hi,

Is delta table maintenance managed by Fabric?

Does Fabric do managed vacuuming of my Lakehouse tables (behind the scenes)?

Or will my Lakehouse delta tables be accumulating all table historical data (version history), until I decide to do a vacuum myself.

Thank you 😀 

[Anonymous]

Hi @frithjof_v 
Thanks for using Fabric Community.

Yes, Microsoft Fabric’s Lakehouse provides a Table Maintenance feature that efficiently manages delta tables. This feature includes several operations:

Optimize:
This operation consolidates multiple small Parquet files into larger ones. Larger files improve compression and data distribution across the cluster nodes, reducing the need to scan numerous small files for efficient read operations.

V-Order: This operation applies optimized sorting, encoding, and compression to Delta Parquet files to enable fast read operations across all the Fabric engines.

Vacuum: This operation removes old files no longer referenced by a Delta table log. Files need to be older than the retention threshold, and the default file retention threshold is seven days. All the delta tables in OneLake have the same retention period. This maintenance is important to optimize storage cost.

So, to answer your question, yes, Fabric does manage vacuuming of your Lakehouse tables. However, it’s important to note that setting a shorter retention period impacts Delta’s time travel capabilities. It’s a general best practice to set a retention interval to at least seven days, because old snapshots and uncommitted files can still be in use by the concurrent table readers and writers. Cleaning up active files with the VACUUM command might lead to reader failures or even table corruption if the uncommitted files are removed.

Therefore, your Lakehouse delta tables will not be accumulating all table historical data (version history) indefinitely. They will be managed by the Table Maintenance feature in Fabric, which includes the Vacuum operation.

For more information please refer to this document:
Use table maintenance feature to manage delta tables in Fabric - Microsoft Fabric | Microsoft Learn

Hope this helps. Please let me know if you have any further questions.

[frithjof_v]

Thank you @Anonymous,

If I don't run any maintenance (more specifically: vacuuming) on the Lakehouse delta tables, will the tables keep accumulating all the data and not delete any data?

So if I don't choose to run vacuuming, then the Lakehouse tables will never be vacuumed?

Or does Fabric run vacuuming of the Lakehouse tables in the background by itself, even if I don't do anything to execute the vacuuming?

I'm trying to understand, regarding Lakehouse, if I have the full control over the retention of the data, or if Fabric will delete old versions even if I don't do anything.

For the data warehouse I think Fabric will automatically delete versions older than 7 days. I don't think I can control that.

Is it correct that there is a difference between Lakehouse and Warehouse when it comes to whether I can control the time travel retention?

Can Lakehouse have time travel with as long retention period as I want (possibly 5 years or even more if I want to), and it only vacuums the tables if and when I tell it to vacuum the tables? So if I don't do anything with respect to vacuuming, the Lakehouse tables will never get vacuumed, and I can time travel through the entire history of the table?

Whereas Data Warehouse will be limited to 7 days time travel, and Fabric automatically performs the vacuuming without me being able to control it?

Thank you 😀

[Anonymous · Accepted solution]

Hi @frithjof_v 

Yes, you’re correct in your understanding of the differences between Lakehouse and Data Warehouse in terms of time travel retention and vacuuming.

Lakehouse:
In Lakehouse, the vacuuming process is part of the Table Maintenance feature. This feature includes the Vacuum operation, which removes old files no longer referenced by a Delta table log. The files need to be older than the retention threshold, and the default file retention threshold is seven days.
Use table maintenance feature to manage delta tables in Fabric - Microsoft Fabric | Microsoft Learn

If you don’t run any maintenance, the Lakehouse tables will keep accumulating all the data until the Vacuum operation is executed. So, if you don’t choose to run vacuuming, then the Lakehouse tables will never be vacuumed. 

Data Warehouse:
In Data Warehouse, the data retention for time travel queries is seven days. This is a limit across the OneLake for the entire Fabric. You can’t change it as of now. Therefore, Data Warehouse will be limited to 7 days time travel, and Fabric automatically performs the vacuuming without you being able to control it.
Time travel in Warehouse within Microsoft Fabric (preview) - Microsoft Fabric | Microsoft Learn

Hope this helps. Please let me know if you have any further questions.

[frithjof_v]

Thank you @Anonymous,

I'm struggling to understand this information: "However, all the delta tables in OneLake have the same retention period."

Let's say I run vacuum on a Lakehouse table. Then I can choose the retention period for that vacuum operation freely. If I run a vacuum operation on another table, then I can choose another retention period for that vacuum operation.

So I would say that all tables don't have the same retention period. The retention period is chosen by me whenever I choose to run a vacuum operation on a single table. Or am I missing something?

Thank you 😀 

[Anonymous]

Hi again @frithjof_v 
Yes you are right. I have edited the above post. The actual retention period for each table can be set individually based on your requirements when you perform a vacuum operation.

Hope this helps.😊