Forum Discussion
Advocate IIDataflow Gen2 Reports Success but Gold Lakehouse Missing Records
We are experiencing an intermittent issue with Microsoft Fabric Dataflows Gen2 when loading data from a Silver Lakehouse table to a Gold Lakehouse table within the same workspace. The Dataflow run always completes with a Succeeded status and no errors, but the data is not consistently transferred. Sometimes the Gold table is fully updated, while other times it contains partial data or no new records, even though the Silver table has the correct data. This behavior is non-deterministic and re-running the same Dataflow may or may not resolve the issue. Fast Copy connectors were initially enabled but are now disabled, and the issue still occurs. We have also ensured there are no parallel writers, added wait times between Silver load and Gold Dataflow execution, verified schema consistency, and validated row counts before and after the load. Despite this, row count mismatches continue to happen while the Dataflow reports success. This is a serious concern for financial / reconciliation workloads, as it results in silent data loss. We would like to understand whether Dataflow Gen2 guarantees snapshot isolation when reading Lakehouse Delta tables, whether there are known read-after-write consistency limitations, and if this is a known issue or limitation. Any recommended best practices for reliable Silver-to-Gold data movement using Dataflows would be greatly appreciated.
This issue is still not fully resolved. However, I have managed it using a notebook. I added validation logic and implemented retry attempts until the table is successfully updated.
9 Replies
- KGU_16031986
Advocate I
Any specific reason for using dataflows Gen2 to load data to silver and gold laters than using data pipelines with copy activity or Stored procedures?
NuwanAthukoralaWe used Dataflows Gen2 because they are positioned by Microsoft as a Lakehouse-native tool for Silver and Gold transformations, supporting data cleansing, standard transformations, schema handling, and governance in a low-code manner. Copy activity is mainly optimized for ingestion, and stored procedures are less suitable for cross-Lakehouse transformations. Based on this positioning, we expected Dataflows Gen2 to be reliable for Silver-to-Gold processing.
- tayloramy
Super User
Hi NuwanAthukorala,
Is the dataflow conneting to delta tables directly, or using the SQL Endpoint?
The SQL Endpoint takes some time to update.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
- NuwanAthukorala
I used Delta tables directly.
V-yubandi-msftCommunity Support
Hi NuwanAthukorala ,
Yes, the Dataflow connects directly to the Lakehouse Delta tables, not through the SQL Endpoint, so SQL refresh delay shouldn't be the cause here.
Our main question is whether Dataflows Gen2 provides a consistent snapshot when reading Delta tables. In our scenario, the Silver load is completed and there are no parallel writers when the Gold Dataflow runs. Still, we observe inconsistent results sometimes all rows are copied, and other times only some rows are read, even though the Dataflow always shows a Succeeded status without errors.
Since this is for financial and reconciliation purposes, these silent partial reads are a significant concern. We would like clarification on
1. Does Dataflows Gen2 guarantee snapshot isolation when reading Delta tables?
2. Are there any known read after write consistency issues?
3. Is this a known issue for Silver to Gold patterns & What best practices ensure reliable movement between Lakehouse tables?Thank You.
- V-yubandi-msft
Hi NuwanAthukorala ,
Could you please confirm whether the issue has been resolved? If you need any further information or support, we’ll be happy to help.
Thank you. - V-yubandi-msft
May I know if your issue is resolved, or if you still need any assistance? Please feel free to let us know if we can
help further.
Thanks,
Yugandhar- NuwanAthukorala
This issue is still not fully resolved. However, I have managed it using a notebook. I added validation logic and implemented retry attempts until the table is successfully updated.
- V-yubandi-msft
Hi NuwanAthukorala ,
Thank you for the update. It’s good to hear you managed this by using a notebook with validation and retry logic. That’s a practical solution, especially when working with financial data that needs complete accuracy.
As you mentioned earlier, Dataflow sometimes showed a Succeeded status even when the Gold table wasn’t fully updated. Your approach with retries and validation helps address that issue.
Thanks for sharing your solution it will be useful for others as well.
