Forum Discussion
Skilled SharerDelta Lake VACUUM, how does it work?
I'm new to Delta Lake, but I think this link explains what the vacuum retention threshold means:
https://docs.databricks.com/en/delta/vacuum.html#example-syntax-for-vacuum
"Use the RETAIN keyword to specify the threshold used to determine if a data file should be removed. The VACUUM command uses this threshold to look back in time the specified amount of time and identify the most recent table version at that moment. Delta retains all data files required to query that table version and all newer table versions."
My interpretation:
When we vacuum a table at 2024-04-16 13:29:00 with a retention threshold of 7 days, then we basically tell the vacuum operation to look back to the timepoint 2024-04-09 13:29:00 and identify what was the most recent table version (the "active" table version) at the timepoint 2024-04-09 13:29:00.In your case, at that timepoint, I think your version 1 was the most recent version of the table. So the parquet files needed to query version 1 of the table will not be deleted.
If there existed any parquet files which were only needed for querying version 0 (but not needed for querying version 1 or more recent versions), then I think those parquet files would have been deleted.I think if you go to the Lakehouse explorer, and click on the ellipsis (...) next to the table name, and then "View files", you should see the actual parquet files which are still existing.
If any parquet files were deleted by the vacuum operation, then I think you will not be able to find those parquet files here anymore.However, I think the version history of your delta table may still contain references to deleted parquet files.
This depends on the log history retention period, if I understand correctly (see next comments).When you use
%%sql DESCRIBE HISTORY table_name;I think the history can still include references to some parquet files which are now deleted, because the history log is not deleted by the vacuum operation (see next comments). However the actual data files (parquet files) in your lakehouse table's file directory should be deleted by vacuum operations (according to vacuum retention threshold).
Community ChampionI'm new to Delta Lake, but I think this link explains what the vacuum retention threshold means:
https://docs.databricks.com/en/delta/vacuum.html#example-syntax-for-vacuum
"Use the RETAIN keyword to specify the threshold used to determine if a data file should be removed. The VACUUM command uses this threshold to look back in time the specified amount of time and identify the most recent table version at that moment. Delta retains all data files required to query that table version and all newer table versions."
My interpretation:
When we vacuum a table at 2024-04-16 13:29:00 with a retention threshold of 7 days, then we basically tell the vacuum operation to look back to the timepoint 2024-04-09 13:29:00 and identify what was the most recent table version (the "active" table version) at the timepoint 2024-04-09 13:29:00.
In your case, at that timepoint, I think your version 1 was the most recent version of the table. So the parquet files needed to query version 1 of the table will not be deleted.
If there existed any parquet files which were only needed for querying version 0 (but not needed for querying version 1 or more recent versions), then I think those parquet files would have been deleted.
I think if you go to the Lakehouse explorer, and click on the ellipsis (...) next to the table name, and then "View files", you should see the actual parquet files which are still existing.
If any parquet files were deleted by the vacuum operation, then I think you will not be able to find those parquet files here anymore.
However, I think the version history of your delta table may still contain references to deleted parquet files.
This depends on the log history retention period, if I understand correctly (see next comments).
When you use
%%sql
DESCRIBE HISTORY table_name;
I think the history can still include references to some parquet files which are now deleted, because the history log is not deleted by the vacuum operation (see next comments). However the actual data files (parquet files) in your lakehouse table's file directory should be deleted by vacuum operations (according to vacuum retention threshold).
- frithjof_v
If some parquet files have been deleted by vacuuming, I think you could get an error if you try to query the table versions which reference the deleted parquet files.
For example, if some parquet files needed to query table version 0 has been deleted, then this query should give an error:
%%sql SELECT * FROM table_name VERSION AS OF 0I think such an error will contain this text:
"It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved."- frithjof_v
Reading this article, it seems that the log history is kept for 30 days by default:
https://docs.databricks.com/en/delta/history.html
And the vacuum retention threshold is 7 days by default.
I haven't tested this, but if I understand this article correctly, then the ability to do time travel is actually limited by both the log history retention and the vacuum operation's retention threshold.
The way I understand it, if you do vacuuming with 7 days retention threshold, you can only do time travel within the last 7 days.
If you don't do vacuuming, I think you can only do time travel within the number of days your log history is kept (default is 30 days).
As the article says:
"Databricks does not recommend using Delta Lake table history as a long-term backup solution for data archival. Databricks recommends using only the past 7 days for time travel operations unless you have set both data and log retention configurations to a larger value."
There seems to be something called checkpoints, which seem to affect the actual time when the log history (older than the log retention period) gets deleted.
https://docs.databricks.com/en/delta/history.html#what-are-transaction-log-checkpointsAnyway, if we do vacuuming with 7 days retention threshold, then I guess vacuuming will be the limiting factor for the time travel, as the log retention period is 30 days by default.
I guess we can adjust the vacuum retention threshold and log retention period, if we want to expand (or limit) the possibility for time travel.
amaaiiaSo, if that's true about checkpoints, how is it possible that I can get VERSION 1 for a table where first checkpoint is done after this version timestamp?
This is the HISTORY of my table. As you can see, version 1 is from 2024-03-09:
Then, I check parquet files and I see there's a checkpoint on 2024-03-14:
As a I've understood, even if the log files still exist, as there's a checkpoint, the previous logs shouldn't be available. So, why can I get VERSION 1 of table data if the first checkpoint points to VERSION 10?
- amaaiia
That's true. I was confused with the version timestamp, VACUUM keeps files for the active version at that time, I thought it kept files of versions with higher timestamp only. I've tried with another table with more versions and It works as you say.