Forum Discussion
Semantic model refresh does not refresh data in report
I think I may have a solution... I've added a step to hit the endpoint with a simple query before refreshing my model, looks like it might do the trick.
You may want to follow my thread on the subject here: https://community.fabric.microsoft.com/t5/Service/Semantic-Model-connected-to-Lakehouse-SQL-Analytics-endpoint-not/m-p/4828558/highlight/false#M286453
Interesting, thanks for the detail!
I don't think timing is an issue in my case, as I'm running the model refresh in a separate pipeline which runs more than an hour after my ingestion pipelines complete.
I see the linked post is using a raw web API call to trigger the refresh, rather than using the specific Semantic Model Refresh activity - I would assume that the specific activity just calls the same API under the hood, but you never know...
In your case, is the semantic model in the same workspace / capacity as the lakehouse?
Hi DuncanKing,
It might be that under the hood it is the same API call, but when I look at the refresh history of the semantic model, I see that the Semantic model refresh activitys Type is Data Factory, while the above mentioned methods Type is API.
In my case I have only one capacity, and the semantic model is in the same workspace as the Lakehouse.
In case I see that this method works consistently my plan is to bring the same pipeline through my test and prod stages as well, but even than the (prod) semantic model will live in the same (prod) workspace as the (prod) Lakehouse.
I am genuinely puzzled.
I am running the above pipeline every day at 6 AM while still out of office, and while the pipeline runs without and error, I consistently get the old data both in the semantic model and in the report.
But then when I get to the office, I change the schedule to maybe 8:10 (or any other random time), let the same pipeline run again and after a successfull pipeline run I see the fresh data in the semantic model and in the reports as well.
In both cases I run a scheduled pipeline, the only difference being that I run it at a different time.
This is a consisteny behaviour and I don't find any logic in this.
Either I do something gravely wrong, Microsoft is not on top of the refresh game or Fabric wants some human company to work.
DuncanKing did you try the above method? If yes what is your experience?I have a meeting next week with a Microsoft Fabric expert and I hope that he can show me a method that consistently works.
If so I will definitely post the solution here.
I think I may have a solution... I've added a step to hit the endpoint with a simple query before refreshing my model, looks like it might do the trick.
You may want to follow my thread on the subject here: https://community.fabric.microsoft.com/t5/Service/Semantic-Model-connected-to-Lakehouse-SQL-Analytics-endpoint-not/m-p/4828558/highlight/false#M286453
Hi DuncanKing,
Your reply is most useful, although I also need time to confirm if it works as expexted.
My gut feeling says that this might be the solution, and your post would also perfectly explain why my 6AM scheduled pipeline run does not update the semantic model, while when I run it later (after I made some queries on the Lakehouse) it does the update.
Now I also added a Lookup activity after the Gold pipeline where I just get the MAX of my TransactionId. In theory this would "warm up" the Lakehouses metadata, and when I do the Semantic model refresh after, it should get the new data.
I only implemented this method this morning though, so I also need a couple of days to confirm the validity of this claim but this also explains why I first got old data displayed when I ran an SQL query directly on my Lakehouse, but when I ran the query again right after it showed the new data; the metadata was not updated because the endpoint was not warm.
Fingers crossed 🤞
P.S.: I wonder if the same applies in case we have the Semantic model refresh activity in the pipeline instead of the API call... 🤔
I haven't had a chance to get near it yet, been super busy with client workshops on other projects!