Forum Discussion
Solved
Microsoft Fabric - Data Pipeline Syncing Issue
Hi, Me and my colleague have noticed an issue with a few pipelines recently. For context the pipelines architecture is made up of 4 dataflows in a linear flow, the first two extract data from...
Hi Jester_3
It is a known behavior that can occur in Fabric when the Lakehouse/OneLake metadata (Delta log & SQL analytics endpoint) hasn’t caught up with the latest write yet. In short: the data is written promptly, but the readers in subsequent steps sometimes hit stale metadata for a few minutes.I think dataflow 3 is using the SQL analytics endpoint internally, which is why it is experiencing this.It is best you introduce an extra step to confirm changes have been written i.e. introduce a “commit confirmation” notebook step between Dataflow 2 and 3Insert a lightweight Notebook that:
- Reads the Delta log/version of your Silver target tables and waits until a new version appears; or
- Performs a harmless SELECT COUNT(*) loop until the expected watermark changes.
This aligns with reports that the Delta log/metadata propagation is the bottleneck; explicit polling avoids running Dataflow 3 while the table is still on the prior version.
Hope this helps, please appreciate by leaving a Kudos or accepting as a Solution!
Hi Jester_3
It is a known behavior that can occur in Fabric when the Lakehouse/OneLake metadata (Delta log & SQL analytics endpoint) hasn’t caught up with the latest write yet. In short: the data is written promptly, but the readers in subsequent steps sometimes hit stale metadata for a few minutes.
I think dataflow 3 is using the SQL analytics endpoint internally, which is why it is experiencing this.
It is best you introduce an extra step to confirm changes have been written i.e. introduce a “commit confirmation” notebook step between Dataflow 2 and 3
Insert a lightweight Notebook that:
- Reads the Delta log/version of your Silver target tables and waits until a new version appears; or
- Performs a harmless SELECT COUNT(*) loop until the expected watermark changes.
This aligns with reports that the Delta log/metadata propagation is the bottleneck; explicit polling avoids running Dataflow 3 while the table is still on the prior version.
Hope this helps, please appreciate by leaving a Kudos or accepting as a Solution!
Hi deborshi_nag
Thanks for this suggested fix, I think I'll have to apply this to all my pipelines currently as this issue is happening more requently this past week.
The long description of the fix may be beneficial for others so i've went into detail below.
I ended up adding an until loop with a timeout of an hour in my pipeline:
The conditional check on the pipeline was :
@equals(
activity('DELTA_CHECK').output.result.exitValue,
formatDateTime(utcNow(), 'yyyy-MM-dd')
)
The notebook content :
from delta.tables import DeltaTable
from datetime import datetime
table_path = "Tables/ABC"
delta_table = DeltaTable.forPath(spark, table_path)
history_df = delta_table.history()
timestamp = str(history_df.select("Timestamp").head()[0])
dt = datetime.fromisoformat(timestamp)
date_only = dt.date()
mssparkutils.notebook.exit(str(date_only))