Forum Discussion
Change Data Feed bug? Unable to reconstruct state despite recent log files
Hello mhseko
Your error is not a bug in Delta Lake or CDF, but a result of how retention and log management work. Once the log or checkpoint files for a version are gone, you cannot reconstruct the table or access its CDF for that version, regardless of visible recent logs or lack of `VACUUM` commands. To avoid this in the future, extract and store CDF data you need to retain before the retention window expires.
Delta log files have a default retention period of 30 days in Fabric, after which they are automatically deleted. This is not controlled by manual `VACUUM` commands; the deletion is managed by Fabric’s background processes
If you want to retain more history, you must set the retention properties (`delta.logRetentionDuration`, `delta.deletedFileRetentionDuration`) to a longer interval. These should persist unless the table is dropped/recreated
Hello mhseko
Your error is not a bug in Delta Lake or CDF, but a result of how retention and log management work. Once the log or checkpoint files for a version are gone, you cannot reconstruct the table or access its CDF for that version, regardless of visible recent logs or lack of `VACUUM` commands. To avoid this in the future, extract and store CDF data you need to retain before the retention window expires.
Delta log files have a default retention period of 30 days in Fabric, after which they are automatically deleted. This is not controlled by manual `VACUUM` commands; the deletion is managed by Fabric’s background processes
If you want to retain more history, you must set the retention properties (`delta.logRetentionDuration`, `delta.deletedFileRetentionDuration`) to a longer interval. These should persist unless the table is dropped/recreated
I have verified this with multiple delta tables. A given transaction log file cannot be utilized if the preceding checkpoint file is missing. Ok, I'll update schedules accordingly. In general, the processes I have work fine. This issue cropped up for a table that did not have updates for an extended period of time.