Forum Discussion

Jester_3's avatar
Jester_3
Frequent Visitor
7 months ago
Solved

Microsoft Fabric - Data Pipeline Syncing Issue

Hi,   Me and my colleague have noticed an issue with a few pipelines recently.   For context the pipelines architecture is made up of 4 dataflows in a linear flow, the first two extract data from...
data engineering
Data Pipeline
  • deborshi_nag's avatar
    deborshi_nag
    7 months ago

    Hi Jester_3 

     

    It is a known behavior that can occur in Fabric when the Lakehouse/OneLake metadata (Delta log & SQL analytics endpoint) hasn’t caught up with the latest write yet. In short: the data is written promptly, but the readers in subsequent steps sometimes hit stale metadata for a few minutes.
     
    I think dataflow 3 is using the SQL analytics endpoint internally, which is why it is experiencing this. 
     
    It is best you introduce an extra step to confirm changes have been written i.e. introduce a “commit confirmation” notebook step between Dataflow 2 and 3
     

    Insert a lightweight Notebook that:

    • Reads the Delta log/version of your Silver target tables and waits until a new version appears; or
    • Performs a harmless SELECT COUNT(*) loop until the expected watermark changes.
      This aligns with reports that the Delta log/metadata propagation is the bottleneck; explicit polling avoids running Dataflow 3 while the table is still on the prior version.
    Hope this helps, please appreciate by leaving a Kudos or accepting as a Solution!