Forum Discussion
Solved
Microsoft Fabric - Data Pipeline Syncing Issue
Hi, Me and my colleague have noticed an issue with a few pipelines recently. For context the pipelines architecture is made up of 4 dataflows in a linear flow, the first two extract data from...
Hi Jester_3
It is a known behavior that can occur in Fabric when the Lakehouse/OneLake metadata (Delta log & SQL analytics endpoint) hasn’t caught up with the latest write yet. In short: the data is written promptly, but the readers in subsequent steps sometimes hit stale metadata for a few minutes.I think dataflow 3 is using the SQL analytics endpoint internally, which is why it is experiencing this.It is best you introduce an extra step to confirm changes have been written i.e. introduce a “commit confirmation” notebook step between Dataflow 2 and 3Insert a lightweight Notebook that:
- Reads the Delta log/version of your Silver target tables and waits until a new version appears; or
- Performs a harmless SELECT COUNT(*) loop until the expected watermark changes.
This aligns with reports that the Delta log/metadata propagation is the bottleneck; explicit polling avoids running Dataflow 3 while the table is still on the prior version.
Hope this helps, please appreciate by leaving a Kudos or accepting as a Solution!
What you’re hitting is the lag between physical writes and metadata visibility in Fabric’s SQL analytics endpoint. Dataflow 3 is likely reading stale metadata even though the Bronze‑to‑Silver write has completed.
Your commit‑check workaround is solid. A few refinements worth noting for others:
- Poll Delta version rather than just the date, so multiple commits in a day don’t collide.
- Tune timeouts — most syncs finish in minutes, so shorter waits with retries can reduce pipeline latency.
In short, explicit synchronization is the safest way to ensure downstream dataflows operate on the latest state until Fabric improves metadata propagation.