Forum Discussion
Continued ContributorSQL Endpoint lakehouse: Intermittent error in the Refresh SQL analytics endpoint activity
Thank you for all the responses, going down the full api/notebook route is too much of an overhead for us at the moment. It would require us to set up a service principal, api keys etc, which we (don't) have (yet) (very small team :-) )
We do know from experience that a 5-minute wait in our pipeline, though ugly, solved the issue. Endpoints would have synched by then.
(The only time that did not work flawless was on 1 pipeline that ran at 130 am in the morning, when the lakehouse was well deep asleep. I solved that by running a dummy script against the endpoint...select top (1) from etc etc.
With that in mind I would think that (in our situation) "on completion" would be safe?
CheersHans
Continued ContributorScheduling the refresh activity to only run after the write to lakehouse is sadly not an option. We have multiple pipelines some relying on each other , some scheduled specifically to meet a Business need. I fear a retry could potentially only make it worse.
Another idea i had was to have the next step in the pipeline start on both success and on fail. Would that work this way? The reason is: The error will be thrown after about 12 minutes, which means that specific dataset in the sql-endpoint will have refreshed anyway?
I mean something like this:
- ShivekMaharaj
Community Champion
Hi smeetsh,
Yes, it is possible that some tables have already synchronized before the activity ultimately fails.
The Refresh SQL analytics endpoint operation works against unsynchronized Lakehouse metadata/tables, so an overall timeout or failure does not necessarily mean that every table remained unchanged.
Fabric also maintains a background metadata synchronization process independently of this activity.
Microsoft documents the activity behavior in the Refresh SQL analytics endpoint guidance.
Because of that, I would be careful with this pattern:
Refresh SQL endpoint -> on success OR failure -> continue with downstream processingIt is technically possible to configure the pipeline that way, but a failure branch means you no longer have a deterministic guarantee that the SQL endpoint reflects all Lakehouse changes.
If the next activity can tolerate slightly stale metadata, then continuing on failure may be acceptable.
If the downstream step requires the SQL endpoint to be fully current, I would instead use:
Refresh SQL endpoint -> retry with delay -> continue only on successand handle the final failure separately.
Microsoft’s pipeline activities support configurable retries and retry intervals, which is useful for these transient timeout/locking scenarios.
There is also one useful distinction:
Success -> unsynchronized data was synchronized successfully NotRun -> there was nothing new to synchronize Failure -> something went wrong, and you should not assume the endpoint is fully synchronizedSo I would not use the failure path as an equivalent success path unless your downstream logic explicitly tolerates eventual consistency.
Given your ~12 minute timeout pattern, I would prefer a delayed retry first, and only continue after failure if the business process can safely work with potentially stale SQL endpoint metadata.
AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.
