Forum Discussion
Semantic model refresh does not refresh data in report
I think I may have a solution... I've added a step to hit the endpoint with a simple query before refreshing my model, looks like it might do the trick.
You may want to follow my thread on the subject here: https://community.fabric.microsoft.com/t5/Service/Semantic-Model-connected-to-Lakehouse-SQL-Analytics-endpoint-not/m-p/4828558/highlight/false#M286453
Hi v-echaithra ,
As i mentioned to rohit1991, i am still conducting my tests and will get back with a feedback on monday.
I want to note though that I had the same problem yesterday, as i described in my original post.
I could confirm that i received the new data into the lakehouse, but this time when i checked the data in my semantic model, it was still showing the old data even after a seemingly successfull scheduled refresh, and of course the same applied to the report as well.
After a manual refresh of the semantic model, my report that i connected to it updated the new data and was working fine, but most interestingly the same semantic model did not show the new data when explored.
To add more context, i read up a little about the semantic model refresh, and ended up appending a semantic model refresh activity to my ETL pipeline instead of using the scheduled refresh directly from the semantic model which i even turned off.
After running my ETL pipeline, i immediately checked my report which was showing the new and updated data. As of now, it seems like using the semantic model refresh activity works instead of the scheduled refresh of the semantic model.
To confirm this statement i still need some time for testing, so i might even close this case later next week when i could test the semantic model refresh activity method more thoroughly.
Thank you for your patience!
Hi Mark
I'm experiencing a very similar issue around refreshing a semantic model from the lakehouse SQL analytics endpoint... Did you get a resolution in the end? You mention that using a SM refresh activity in your pipeline might work better - but I've tried this with without success. Did it work properly, and if so, could you give any more detail?
Cheers
D
Hi DuncanKing,
Thanks for reaching out.
I might have a solution now, but I need to verify its validity. It worked today at least...but I would need to confirm if this works consistently.
On the other hand it would be nice to see if this method works with others as well.
Here it goes.So I restructured my ETL pipeline first.
Previously I had one main pipeline that was calling different notebooks for my bronze, silver and gold transformations, and connected the semantic model refresh activity after it. I am guessing that right after the last gold transformation notebook where I also save the data to my Lakehouse, there was not enough time for the Lakehouse metadata to be changed, and so the Semantic model refresh activity copied the old data.
So,
1. I created child pipelines:
- the bronze pipeline only held the copy activities from source and a bronze notebook (which calls an OAuth2 API, that I could not solve with any other activity, but this is a completely different issue)
- a silver pipeline for the silver transformation notebooks
- a gold pipeline for the gold transformations
2. I created a parent pipeline that calls the bronze, silver and gold pipelines after each other.
3. I created yet another pipeline that does the semantic model refresh. I followed the exact steps from this article to create this semantic model refresh pipeline: https://medium.com/@gordeyew/refresh-semantic-model-with-microsoft-fabric-data-pipeline-4f5e7f6c479b
4. I added the semantic model refresh pipeline after my gold pipeline (on success), AND made sure that the Wait on completion option was turned on on the Gold pipeline.
Now my parent pipeline looks like this:
NOTE:1. I am also trying to implement CI/CD principles, and so far I only ran this pipeline in my dev stage as seen on the workspaces option.
2.
- This morning I checked if the pipeline ran successfully, which it did.- Then I checked if I have the latest data in my Lakehouse, which it did.
- Then I checked the semantic model if it was updated in the Refreshed column, which showed todays date.
- Then I clicked the explore semantic model option and checked the transactionId column value to see if it is the same as in the Lakehouse, and I,ve seen the latest data.
- Finally I've opened my report that was showing the old data, but when I refreshed the visual the report showed todays data.
3. This method needs to be validated to see if it is reliable or not. Verification will take me about a week or so...
I will keep this post updated about my experience, and if you also end up trying this, please leave a feedback on your findings, it would be super helpful.
Interesting, thanks for the detail!
I don't think timing is an issue in my case, as I'm running the model refresh in a separate pipeline which runs more than an hour after my ingestion pipelines complete.
I see the linked post is using a raw web API call to trigger the refresh, rather than using the specific Semantic Model Refresh activity - I would assume that the specific activity just calls the same API under the hood, but you never know...
In your case, is the semantic model in the same workspace / capacity as the lakehouse?
Hi DuncanKing,
It might be that under the hood it is the same API call, but when I look at the refresh history of the semantic model, I see that the Semantic model refresh activitys Type is Data Factory, while the above mentioned methods Type is API.
In my case I have only one capacity, and the semantic model is in the same workspace as the Lakehouse.
In case I see that this method works consistently my plan is to bring the same pipeline through my test and prod stages as well, but even than the (prod) semantic model will live in the same (prod) workspace as the (prod) Lakehouse.