Forum Discussion
Delta table delays
- 1 year ago
Hi ray_codex ,
Thank you for reaching out to Microsoft Fabric Community.
The delays in Delta table availability you are experiencing are likely due to:
-
Eventual consistency in Lakehouse file storage and delta transaction logs.
-
Latency in Dataflow Gen2 writing the files and committing the delta transactions.
-
The dbo schema being the default dump location, which may not immediately reflect new files when accessed programmatically.
-
Lakehouse metadata especially for delta tables sometimes taking time to update even though the file is physically present.
Please refer these documents for more clarification-
https://learn.microsoft.com/en-us/azure/data-factory/format-delta
https://learn.microsoft.com/en-us/azure/data-factory/connector-microsoft-fabric-lakehouse
https://learn.microsoft.com/en-us/azure/data-factory/connector-microsoft-fabric-lakehouse
If my response has addressed your query, please Accept as Solution and give a 'Kudos' so other members can easily find it.
Thank You -
Hi ray_codex ,
Thank you for reaching out to Microsoft Fabric Community.
The delays in Delta table availability you are experiencing are likely due to:
-
Eventual consistency in Lakehouse file storage and delta transaction logs.
-
Latency in Dataflow Gen2 writing the files and committing the delta transactions.
-
The dbo schema being the default dump location, which may not immediately reflect new files when accessed programmatically.
-
Lakehouse metadata especially for delta tables sometimes taking time to update even though the file is physically present.
Please refer these documents for more clarification-
https://learn.microsoft.com/en-us/azure/data-factory/format-delta
https://learn.microsoft.com/en-us/azure/data-factory/connector-microsoft-fabric-lakehouse
https://learn.microsoft.com/en-us/azure/data-factory/connector-microsoft-fabric-lakehouse
If my response has addressed your query, please Accept as Solution and give a 'Kudos' so other members can easily find it.
Thank You