Lakehouse Data Not Reflected via SQL Analytics Endpoint - Solution 1

[odtJitendra]

Problem Description

Every Fabric Lakehouse automatically provisions a SQL Analytics Endpoint (SAE) that allows the data written into Delta tables to be queried using T-SQL. In our pipelines, we write data into a Lakehouse (via notebooks, Dataflows Gen2, or pipeline Copy activities) and then immediately trigger a downstream activity — typically a Copy or Stored Procedure activity — that reads from the Lakehouse through the SQL Analytics Endpoint to insert that data into a Fabric Warehouse. We have repeatedly observed that when the downstream activity runs immediately after the Lakehouse write, it either inserts stale (old) data, misses newly added rows, or fails to pick up schema changes such as new/renamed columns. The Warehouse ends up out of sync with the Lakehouse even though the pipeline reports success on every activity.

Workaround A — Force an on-demand metadata sync before reading

Before the downstream Warehouse-load activity runs, explicitly trigger a metadata refresh on the SQL Analytics Endpoint (via the Fabric REST API, or the 'Refresh' action in the SAE explorer) and poll until the sync status reports complete, rather than assuming the data is ready.

Anyone facing the same ?

[tayloramy]

Hi @odtJitendra, 

This is known behaviour. After loading data to a lakehouse, I recommend you do a metadata sync, either though the REST API, or using the Pipeline activity to ensure that your SQL Endpoint is up to date.  

[v-aatheeque]

Hi @odtJitendra 

Yes, this behavior has been observed by multiple users. The SQL Analytics Endpoint (SAE) does not always synchronize immediately after data is written into a Lakehouse. Even though the upstream write activity completes successfully, there can be a short delay before newly added rows, schema changes, or metadata updates are reflected in the SAE. As a result, downstream T‑SQL operations may occasionally read stale data or miss schema changes if they execute right after the write.

Your workaround of triggering an explicit SQL Analytics Endpoint metadata refresh and waiting for the synchronization to complete before running the downstream activity is a valid approach. Alternatively, adding a retry mechanism or a short delay before querying the endpoint can also help mitigate the issue.

If this behavior continues to impact your production workloads despite these workarounds, it's recommended to open a Microsoft Fabric Support and Status | Microsoft Fabric  so the Fabric team can investigate whether the synchronization delay is longer than expected for your environment.

Useful Links : SQL Analytics Endpoint Metadata Sync - Microsoft Fabric | Microsoft Learn