Currently, when creating a shortcut in Fabric to ADLS Gen2 containing delta table, it does not work properly when checkpointPolicy configuration is set to v2.
Reference: https://github.com/delta-io/delta/blob/master/PROTOCOL.md#v2-spec
From time to time it fails with an error: Delta table 'X' with '_last_checkpoint' file references version 'X' but the associated checkpoint file is missing from the _delta_log directory.
But it is not missing, it just stored in the sidecars folder which is a new addition in v2.
5 Comments
- It would bring much more stability when using Unity catalog mirroring or when you have shortcut tables on top of delta tables created by databricks.
- As it stands now, tables created with default settings in Databricks cannot be read with catalog mirroring in PowerBI/Fabric. This makes this very confusing when setting this up, as Fabric does not tell you what the expectations are for data to be readable.
tabinaAdvocate I
Databricks promotes v2 checkpoints and liquid clustering as best practices, yet Fabric’s SQL endpoint does not support them. It's high time support for v2 checkpoint policy in Delta table shortcuts is enabled so Fabric can align with Databricks standards and eliminate fragile workarounds.- It is kinda strange that Databricks mirroring is GA in Fabric but it cannot handle the standard Databricks configuration. Please implement this ASAP.
- 77 This problem is preventing us from applying data-driven strategies in the company where I work. Please implement this ASAP.
