Delta Table: log .json files not getting cleaned up after checkpoint creation

Hi frithjof_v 

I didn't find any more detailed or helpful documentation on this "logRetentionDuration" property other than the one you've already read. I tried the following commands but it told me the delta table doesn't have "logRetentionDuration" property no matter I enable or disable "retentionDurationCheck" property in Spark configuration. This is probably why it didn't remove the log files. 

From the documentation you linked, it says "Each time a checkpoint is written, Delta automatically cleans up log entries older than the retention interval. If you set this config to a large enough value, many log entries are retained. This should not impact performance as operations against the log are constant time. Operations on history are parallel but will become more expensive as the log size increases." 

It seems the performance of read operations won't be affected significantly when checkpoints exist. A possible impact may be that when there are more and more log files, they will occupy more storage space.

Here are some blogs that may be helpful:

Microsoft Fabric Table Maintenance - Checkpoint and Statistics (mssqltips.com)

Microsoft Fabric Lakehouse OPTIMIZE and VACUUM for Table Maintenance (mssqltips.com)

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!