Forum Discussion
Resolver IIDelta table delays
Hi,
Is anyone having trouble with delays in availability of Delta tables in Lakehouse? I have a pipeline that basically does this:
- Get Excel files from Sharepoint with Dataflow Gen 2
- Sink files to Lakehouse dbo schema (because I have a default destination, many files, cant set schema)
- Copy files from dbo to excel schema (with Notebook, spark SQL)
- drop all tables in dbo schema (with Notebook,PySpark)
I'm noticing reliability issues. I have wait activities in my pipeline, but I still notice sometimes the copy files is running before the actual files are present in lakehouse.
Anyone else having these issues? I've seen people complaining about SQL-Endpoint, but I'm not using the endpoint.
Hi ray_codex ,
Thank you for reaching out to Microsoft Fabric Community.
The delays in Delta table availability you are experiencing are likely due to:
-
Eventual consistency in Lakehouse file storage and delta transaction logs.
-
Latency in Dataflow Gen2 writing the files and committing the delta transactions.
-
The dbo schema being the default dump location, which may not immediately reflect new files when accessed programmatically.
-
Lakehouse metadata especially for delta tables sometimes taking time to update even though the file is physically present.
Please refer these documents for more clarification-
https://learn.microsoft.com/en-us/azure/data-factory/format-delta
https://learn.microsoft.com/en-us/azure/data-factory/connector-microsoft-fabric-lakehouse
https://learn.microsoft.com/en-us/azure/data-factory/connector-microsoft-fabric-lakehouse
If my response has addressed your query, please Accept as Solution and give a 'Kudos' so other members can easily find it.
Thank You-
4 Replies
v-sdhruvCommunity Support
Hi ray_codex ,
Thank you for reaching out to Microsoft Fabric Community.
The delays in Delta table availability you are experiencing are likely due to:
-
Eventual consistency in Lakehouse file storage and delta transaction logs.
-
Latency in Dataflow Gen2 writing the files and committing the delta transactions.
-
The dbo schema being the default dump location, which may not immediately reflect new files when accessed programmatically.
-
Lakehouse metadata especially for delta tables sometimes taking time to update even though the file is physically present.
Please refer these documents for more clarification-
https://learn.microsoft.com/en-us/azure/data-factory/format-delta
https://learn.microsoft.com/en-us/azure/data-factory/connector-microsoft-fabric-lakehouse
https://learn.microsoft.com/en-us/azure/data-factory/connector-microsoft-fabric-lakehouse
If my response has addressed your query, please Accept as Solution and give a 'Kudos' so other members can easily find it.
Thank You-
- burakkaragoz
Super User
Hi ray_codex ,
If your Microsoft Fabric capacity is paused, you won’t be charged for compute usage. However, if you're on a reserved capacity plan, you may still incur charges for the reservation itself.
Please check your Azure billing details or contact Microsoft support for confirmation.
- ray_codex
I dont quite understand your answer, it hasn't got anything to do with capacity pausing, because this is during trail F64.
- burakkaragoz
Thanks for the clarification! Since you're on a trial F64 capacity, you're right—reserved capacity billing doesn't apply in this case.
During the trial period, pausing the capacity typically just stops compute resources, but you shouldn't incur any charges regardless of whether it's paused or running. The original answer was more relevant to paid/reserved plans—apologies for the confusion.
Let me know if you're seeing unexpected behavior or charges during the trial—happy to help dig deeper!
