SQL Endpoint lakehouse: Intermittent error in the Refresh SQL analytics endpoint activity

Hi smeetsh​ ,

Thank you for reaching out to fabric community.

• This is most likely a transient Fabric API timeout not an F8 capacity issue.
• Add a Retry policy with a delay to the Refresh SQL analytics endpoint activity.
• Ensure the activity runs after all Lakehouse write / ingestion operations are completed to avoid concurrent update / locking issues.
• Microsoft's current documentation specifically states that the activity can intermittently fail when other processes are actively updating the underlying Lakehouse, because the SQL analytics endpoint needs to acquire internal locks. However, the exact 408 Request Timeout / A task was canceled error in your example is not explicitly documented as the known issue.

Refresh SQL analytics endpoint activity - Microsoft Fabric | Microsoft Learn

Thanks!!

Hi smeetsh​,

I think there are two possibilities worth separating here.

Microsoft now explicitly documents that the Refresh SQL analytics endpoint activity can fail intermittently when another process is actively writing to or updating the underlying Lakehouse.

The reason is that the SQL analytics endpoint needs to acquire internal locks during the metadata refresh. If ingestion, a notebook, maintenance operation, or another write is holding the relevant data/metadata structures, the refresh can time out.

Microsoft documents that behavior in the Refresh SQL analytics endpoint activity guidance.

So I would first make sure your pipeline follows this pattern:

all ingestion / notebook / maintenance activity
-> wait until those writes are fully complete
-> one Refresh SQL analytics endpoint activity at the end

rather than refreshing multiple times while Lakehouse updates are still happening.

However, I would not say the lock scenario is definitely the explanation for your exact error.

Your message is:

HTTP 408 RequestTimeout
“A task was canceled”

and you are also seeing intermittent success while there is a Fabric service degradation.

That could still be a transient Fabric API/service-side timeout rather than purely Lakehouse lock contention.

A couple of things I would check:

1. Confirm that no notebook, Copy activity, maintenance task, or other writer is still touching the Lakehouse when the refresh starts.
2. Keep the retry, but use a delay/backoff rather than retrying immediately.
3. Check whether failures correlate with periods of concurrent Lakehouse writes.
4. Compare the failed timestamps with Fabric Service Health.
5. If the same 408 occurs when the Lakehouse has been completely idle for several minutes, I would capture the pipeline run ID, workspace ID, endpoint ID, timestamp and correlation/request details and raise a support case.

One other consideration is whether you need the forced refresh on every run.

Microsoft’s SQL analytics endpoint metadata sync documentation says Fabric already maintains the endpoint through a background metadata-sync process.

Under normal operating conditions, Microsoft says the lag between the Lakehouse and SQL analytics endpoint is generally less than one minute, although it can range from seconds to minutes.

So if your downstream workload can tolerate that delay, you may not need to force a refresh after every pipeline execution.

If deterministic freshness is required, then I would keep the activity, but run it once at the very end and retain the retry/backoff logic.

So my view would be:

concurrent Lakehouse writes
-> documented intermittent refresh failures

idle Lakehouse + repeated HTTP 408 failures
-> more likely worth investigating as a Fabric service/API issue

Given that you are already seeing a service degradation, I would not rule out the second case yet.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

Scheduling the refresh activity to only run after the write to lakehouse is sadly not an option. We have multiple pipelines some relying on each other , some scheduled specifically to meet a Business need.  I fear a retry could potentially only make it worse.

Another idea i had was to have the next step in the pipeline start on both success and on fail. Would that work this way? The reason is: The error will be thrown after about 12 minutes, which means that specific dataset in the sql-endpoint will have refreshed anyway?

I mean something like this:

Hi smeetsh​,

On your continue-on-failure idea, one option that gives you a deterministic answer instead of guessing is to call the refresh API directly rather than through the pipeline activity, because the API returns a status per table.

POST https://api.fabric.microsoft.com/v1/workspaces/{workspaceId}
     /sqlEndpoints/{sqlEndpointId}/refreshMetadata

Two optional body parameters in the Refresh SQL Endpoint Metadata API help with your exact situation:

tables   -> refresh only the tables this pipeline wrote
            (up to 25 per call)
timeout  -> defaults to 15 minutes, can be raised

The response lists each table with a status (Success, Failure or NotRun) and its lastSuccessfulSyncDateTime.

A minimal notebook version, which you can call from the pipeline in place of the activity:

import time, requests

ws_id, ep_id = "<workspaceId>", "<sqlEndpointId>"
token = notebookutils.credentials.getToken("pbi")
h = {"Authorization": f"Bearer {token}"}
url = (f"https://api.fabric.microsoft.com/v1/workspaces/{ws_id}"
       f"/sqlEndpoints/{ep_id}/refreshMetadata")

body = {
    "tables": [{"schema": "dbo",
                "tableNames": ["trev_checklist_data"]}],
    "timeout": {"timeUnit": "Minutes", "value": 30},
}

r = requests.post(url, headers=h, json=body)
if r.status_code == 202:  # long-running operation
    op_url = r.headers["Location"]
    while True:
        time.sleep(int(r.headers.get("Retry-After", 15)))
        r = requests.get(op_url, headers=h)
        if r.json().get("status") in ("Succeeded", "Failed"):
            break
    r = requests.get(op_url + "/result", headers=h)

statuses = r.json()["value"]
failed = [t["tableName"] for t in statuses
          if t["status"] != "Success"]
notebookutils.notebook.exit(
    "OK" if not failed else "FAILED:" + ",".join(failed))

Then in the pipeline:

Notebook (scoped refresh)
-> If Condition on the exit value
-> OK      : continue to Copy data
-> FAILED  : Fail activity with the table names

That gives you the "continue anyway" behavior you described, but only when the tables you actually depend on have synced, rather than assuming they did because the activity ran for 12 minutes.

Scoping each call to a few tables also means a pipeline is no longer asking for a full endpoint sync while other pipelines are still writing to the Lakehouse, which should reduce the lock contention behind these timeouts.

So my overall approach would be:

scoped refresh per pipeline
-> per-table status check
-> continue only when the tables you need are synced
-> short retry with delay for genuine transient errors

That keeps your pipelines independent without giving up the guarantee that downstream steps read current data.

AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.