Forum Discussion
Delayed data refresh in SQL Analytical Endpoint
So my info from Microsoft is that it should take a few seconds to a minute for the SQL Endpoint to discover the updated delta logs. They did have an issue which was fixed apparently.
If your issue persists it would be worth logging that with MS support and give them you workspace ID, lakehouse id, and time it was happening
I also load all my data using Synapse notebooks, using overwrite mode. When I query the table in the SQL endpoint in Fabric the data is updated within seconds from when the underlying delta table is updated most of the time - but sometime it takes up to a few minutes for the SQL endpoint to reflect the change in data.
This (somewhat short) delay is what messes with my API triggered dataset refreshes. I would assume that when my activity to load the delta lake table is done, its safe to run the dataset refresh. But this is not always the case. Whenever data has failed to appear in the dataset I always go right back to the SQL endpoint and query the table, at which point I can always see the new/todays data there
I have added a "5 minute wait" activity in my pipeline, and it now works "most of the time", but still sometimes fail to fetch new data, leading me to believe that 5 minutes is not enough in all cases. I will extend this wait activity to 10-15 minutes, to see if this will give the SQL endpoint sufficient time to do "whatever needs to be done" to refresh the shortcuts/links.
But as far as you know, the SQL endpoint should be always-up-to-date with the underlying data? There is no "reframing" activities going on behind the scenes, where it holds a cache of which parquet files makes up the latest verison of the Delta Lake table?
There certainly is reframing with power bi datasets in terms of updating metadata to understand which is the latest delta data. But for the SQL Endpoint I am not aware of any reframing process, it should read the latest transaction log and display the latest data. It's also the fact that it works "some of the time" in your scenario that's troubling me.
What's the volume of data?