Forum Discussion
Delayed data refresh in SQL Analytical Endpoint
So my info from Microsoft is that it should take a few seconds to a minute for the SQL Endpoint to discover the updated delta logs. They did have an issue which was fixed apparently.
If your issue persists it would be worth logging that with MS support and give them you workspace ID, lakehouse id, and time it was happening
Yes, this is troubling for me to.
Most affected tables are rather small, around 100~ MB in total parquet size and 2~ Mil records.
I can easily reproduce the SQL endpoint delay issue by starting a notebook and performing a spark sql update against a delta lake table (update xxx set yyy = current_timestamp() , and in paralell checking the SQL endpoint with a query (select max(yyy) from xxx).
From the moment my spark update query returns "success" it takes anywhere from a few seconds up to a few minutes until i see my data change in the SQL endpoint (The old timestamp is returned for a number of T-SQL refreshes, until the new timestmap is finally returned).
I would expect to not ever be able to see the old timestamp if i start querying through the SQL endpoint after the spark sql query has finished - but this is not the case.
So my info from Microsoft is that it should take a few seconds to a minute for the SQL Endpoint to discover the updated delta logs. They did have an issue which was fixed apparently.
If your issue persists it would be worth logging that with MS support and give them you workspace ID, lakehouse id, and time it was happening
That you for looking into this.
These few minutes delay would explain what I am seeing, and why my API triggered Power BI dataset refreshes fails to fetch new data.. I have had my Power BI refreshes running with a 10 minute delay for a few days now, and everything has been working so far. A 5 minute delay does not seem to be sufficient even now - I "sometime" miss data using only 5 minutes.
It's a real shame that there is "any" delay here though, since I cant be 100% sure "when" my data is available for pushing to Power BI via the endpoint. Its especially bad since this did not appear to be the case in Synapse Serverless.
I have some spark jobs loading data to some of my tables every 30 minutes (and running for 30~minutes), after which an incremental dataset load is triggered via the same pipeline running the notebooks. Having to add a 10 min wait actitvy in my pipeline here will in have a 30% negative effect on the loading times of my jobs (from job start, to when business users see data in Power BI).
I guess the long term solution here would be to use Direct Lake Semantic Models in Power BI, instead of SQL endpoint. This functionality is however still missing some ctirical features for me to be able to use right now (or I would have to heavily remodel my underlying Gold layer to handle the shortcomings of Direct Lake..)
For now though, I am satisfied in knowing why this is happening. Thanks.
I too have this issue, and I think it is a real drawback. Don't understand why there would be a delay in this update between delta tables and the SQL Endpoint. My issue is that if I create a delta table in the Lakehouse, it is not query-able in the SQL Endpoint (even after several minutes), with the error Invalid object name 'lakehousename.dbo.tablename'. Really frustrating.
Hi!
I'm experiencing the same delay with Fabric & SQL Analytics endpoint. Or, rather the users experience it.
When they get back to me I can see that the data is updated in the SQL Views but not in the semantic model. If I refresh the Semantic Model manually it will show the correct value. But no pipelines has updated any data in onelake between the automatic refesh and my manual refresh.
I'm doing all the data processing in Fabric pipelines and notebooks (no Synapse or Links to ADLS) but I still experience the same thing.
I'll try to add a delay between copying the data from Silver to Gold and the refesh of the semantic model to see if it has any effect. Not the greatest of solutions when refresh time is important, but that's life :).
