Forum Discussion
Delta Table time travel not readable
Thank you for sharing the screenshot. When I refer to an unbroken sequence, I mean that the transaction log versions should increase consecutively without any gaps, such as 3, 4, 5, 6, etc. The sequence in your screenshot appears consistent.
Missing files like 00000000000000000000.json, 00000000000000000001.json, and 00000000000000000002.json are not necessarily a problem, since Delta can remove older log files based on log retention settings.
With a checkpoint at version 10, Spark may only be able to reconstruct the table starting from that checkpoint if earlier logs needed for version 9 are no longer available. In this case, time travel would start from version 10.
Thanks....
Thank you for checking and confirming that DESCRIBE HISTORY still displays versions below 10.
This suggests the issue could be related to how Delta reconstructs the table state from the transaction logs. Although earlier versions are listed in the history, time travel needs a complete and continuous transaction log chain starting from a valid checkpoint. If the checkpoint starts at version 10, or if some earlier JSON log files are missing or unreadable, Spark may not be able to reconstruct version 9.
It could be helpful to look at the _delta_log folder to see if a checkpoint file starts at version 10, and to confirm that the JSON log files before version 10 are available and form an unbroken sequence.
Thank You.
Hi V-yubandi-msft,
thanks for your input. And indeed there is a checkpoint at 10.
When you say "form an unbroken sequence", what do you mean by that?
Is it as shown in the image ment to be "broken" as there is no JSON 1 and 2?
Thanks!
Thank you for sharing the screenshot. When I refer to an unbroken sequence, I mean that the transaction log versions should increase consecutively without any gaps, such as 3, 4, 5, 6, etc. The sequence in your screenshot appears consistent.
Missing files like 00000000000000000000.json, 00000000000000000001.json, and 00000000000000000002.json are not necessarily a problem, since Delta can remove older log files based on log retention settings.
With a checkpoint at version 10, Spark may only be able to reconstruct the table starting from that checkpoint if earlier logs needed for version 9 are no longer available. In this case, time travel would start from version 10.
Thanks....