Lakehouse Data Not Reflected via SQL Analytics Endpoint - Solution 1

Hi odtJitendra,

Yes, I have seen this pattern as well. The upstream Lakehouse write can complete successfully before the SQL analytics endpoint has finished synchronizing the latest Delta metadata and data state.

For a production pipeline, I would avoid relying on an arbitrary delay where possible. I would structure the orchestration as:

1. Write or transform the Lakehouse table.
2. Run the Refresh SQL Endpoint activity against that Lakehouse SQL analytics endpoint.
3. Continue only after the refresh activity completes successfully.
4. Then run the Copy activity that reads from the SQL analytics endpoint into the Warehouse.

The Refresh SQL Endpoint activity is intended for exactly this type of dependency, where a downstream step must see a deterministic Lakehouse state rather than waiting for the normal background synchronization.

I would also distinguish between two cases:

• Schema or metadata changes, such as new columns, removed columns or new tables: refresh the SQL analytics endpoint metadata before the downstream read.
• Data-only changes to an existing table: these should normally become visible automatically, but an explicit refresh still gives the pipeline a controlled synchronization point when immediate consistency is required.

A few additional checks may help if the refresh still does not expose the expected state:

• Confirm that the write transaction has fully committed before the refresh starts.
• Make sure the source is a registered Delta table under the Lakehouse Tables area, rather than only Parquet files under Files.
• Check the Refresh SQL Endpoint activity output. A successful pipeline activity and a successful endpoint synchronization are related but not identical; the activity output indicates whether unsynchronized changes were actually processed.
• Avoid running the refresh while another notebook or ingestion process is still writing to the same table, because concurrent writes can cause an intermittent refresh failure.
• For troubleshooting, compare the row count and schema directly in the Lakehouse and SQL analytics endpoint before starting the Warehouse copy.

If you are using the REST API instead of the pipeline activity, I would also wait for the long-running refresh operation to reach a terminal success state rather than treating the initial API response as proof that synchronization has completed.

So your workaround is valid, but I would formalize it as an explicit pipeline dependency:

Lakehouse write → Refresh SQL Endpoint → Warehouse load

That is more reliable than assuming the SQL endpoint is immediately consistent or adding a fixed sleep interval.