Forum Discussion
Delta Lake VACUUM, how does it work?
I'm new to Delta Lake, but I think this link explains what the vacuum retention threshold means:
https://docs.databricks.com/en/delta/vacuum.html#example-syntax-for-vacuum
"Use the RETAIN keyword to specify the threshold used to determine if a data file should be removed. The VACUUM command uses this threshold to look back in time the specified amount of time and identify the most recent table version at that moment. Delta retains all data files required to query that table version and all newer table versions."
My interpretation:
When we vacuum a table at 2024-04-16 13:29:00 with a retention threshold of 7 days, then we basically tell the vacuum operation to look back to the timepoint 2024-04-09 13:29:00 and identify what was the most recent table version (the "active" table version) at the timepoint 2024-04-09 13:29:00.In your case, at that timepoint, I think your version 1 was the most recent version of the table. So the parquet files needed to query version 1 of the table will not be deleted.
If there existed any parquet files which were only needed for querying version 0 (but not needed for querying version 1 or more recent versions), then I think those parquet files would have been deleted.I think if you go to the Lakehouse explorer, and click on the ellipsis (...) next to the table name, and then "View files", you should see the actual parquet files which are still existing.
If any parquet files were deleted by the vacuum operation, then I think you will not be able to find those parquet files here anymore.However, I think the version history of your delta table may still contain references to deleted parquet files.
This depends on the log history retention period, if I understand correctly (see next comments).When you use
%%sql DESCRIBE HISTORY table_name;I think the history can still include references to some parquet files which are now deleted, because the history log is not deleted by the vacuum operation (see next comments). However the actual data files (parquet files) in your lakehouse table's file directory should be deleted by vacuum operations (according to vacuum retention threshold).
Reading this article, it seems that the log history is kept for 30 days by default:
https://docs.databricks.com/en/delta/history.html
And the vacuum retention threshold is 7 days by default.
I haven't tested this, but if I understand this article correctly, then the ability to do time travel is actually limited by both the log history retention and the vacuum operation's retention threshold.
The way I understand it, if you do vacuuming with 7 days retention threshold, you can only do time travel within the last 7 days.
If you don't do vacuuming, I think you can only do time travel within the number of days your log history is kept (default is 30 days).
As the article says:
"Databricks does not recommend using Delta Lake table history as a long-term backup solution for data archival. Databricks recommends using only the past 7 days for time travel operations unless you have set both data and log retention configurations to a larger value."
There seems to be something called checkpoints, which seem to affect the actual time when the log history (older than the log retention period) gets deleted.
https://docs.databricks.com/en/delta/history.html#what-are-transaction-log-checkpoints
Anyway, if we do vacuuming with 7 days retention threshold, then I guess vacuuming will be the limiting factor for the time travel, as the log retention period is 30 days by default.
I guess we can adjust the vacuum retention threshold and log retention period, if we want to expand (or limit) the possibility for time travel.
So, if that's true about checkpoints, how is it possible that I can get VERSION 1 for a table where first checkpoint is done after this version timestamp?
This is the HISTORY of my table. As you can see, version 1 is from 2024-03-09:
Then, I check parquet files and I see there's a checkpoint on 2024-03-14:
As a I've understood, even if the log files still exist, as there's a checkpoint, the previous logs shouldn't be available. So, why can I get VERSION 1 of table data if the first checkpoint points to VERSION 10?
I'm not sure, but I think the deletion of log history happens at the time when the checkpoint is created, and we need to take the log retention period into account.
It seems this checkpoint was created at March 14.
So I think, when this checkpoint was created on March 14, any log history which is older than mid February (March 14 minus 30 days) would be deleted (provided the log retention period is 30 days).
When a new checkpoint is created, it will delete log history older than 30 days counting from the time when the checkpoint gets created.
If you want all log history before the checkpoint to be deleted, then I think you need to set log retention period equal to 0.
--
On a side note: I'm not entirely sure if we can force a checkpoint to be created, or if the creation of checkpoints is managed entirely by the delta lake for us. I am guessing that if you make frequent changes to your table (meaning more frequent delta log files) then the checkpoints will also be created more frequently.
Because the checkpoints optimize query speed, the checkpoints makes it possible for the engine to avoid having to traverse too many delta log files at query time. At this moment I am just guessing.
--
I think it is something like that. But I'm not sure. Will be great if someone can confirm 😀
Hi,
I've opened new thread to discuss it.