Forum Discussion
Regularily updating fact data refresh time
I have created some extensive pipeplines for copying data from our on prem SQL databases. I am doing some incremental loading, but also doing straight overwriting for some fairly large fact tables (a few million rows). I have noticed that if I have a power bi report connecting to the semantic model, after the data load is "complete" in the pipeline it can take quite a while before Power BI can read any data. I get messages saying it can't connect to the Semantic Model, please try again or contact your administrator. This can take up to an hour before the data is truly available. I am guessing because this is because behind the scenes the data is replicating, over perhaps indexes are being rebuilt? Is there any metadata somewhere that tells me when a refresh is truly "done" and available? Thanks.
What you are seeing is expected behaviour in fabric/pbi and it is not just “data copy complete- ready to query”.
After your pipeline finishes, several backend operations still run before the semantic model becomes queryable: data commit to OneLake, delta/columnstore optimization, metadata synchronization, and most importantly semantic model refresh processing (including partition processing and index encoding). During this window, the model can temporarily reject connections, which is why users see “cannot connect”.
There is no single flag in the pipeline that guarantees “PBI ready”. Instead, you need to track semantic model refresh completion, not pipeline completion.
What to use:
-
Monitor semantic model refresh status in Fabric (or Power BI service) this is the source of truth
-
Use refresh history/REST API (Get Refresh Execution Details) to confirm status = Completed
-
If using pipelines, trigger and wait on a dataset refresh activity, not just data load
-
For large overwrite loads, consider incremental refresh/partitioning to avoid full reprocessing delays
Your delay is coming from semantic model processing, not data movement. Treat dataset refresh completion as the real “done” signal, otherwise you will always see this lag.
-
4 Replies
What you are seeing is expected behaviour in fabric/pbi and it is not just “data copy complete- ready to query”.
After your pipeline finishes, several backend operations still run before the semantic model becomes queryable: data commit to OneLake, delta/columnstore optimization, metadata synchronization, and most importantly semantic model refresh processing (including partition processing and index encoding). During this window, the model can temporarily reject connections, which is why users see “cannot connect”.
There is no single flag in the pipeline that guarantees “PBI ready”. Instead, you need to track semantic model refresh completion, not pipeline completion.
What to use:
-
Monitor semantic model refresh status in Fabric (or Power BI service) this is the source of truth
-
Use refresh history/REST API (Get Refresh Execution Details) to confirm status = Completed
-
If using pipelines, trigger and wait on a dataset refresh activity, not just data load
-
For large overwrite loads, consider incremental refresh/partitioning to avoid full reprocessing delays
Your delay is coming from semantic model processing, not data movement. Treat dataset refresh completion as the real “done” signal, otherwise you will always see this lag.
-
Hi alloowishus I think from your description somthing really important is missing, you mentioned from on-prem sql into your semantic model, but what is feeding semantic model? I'm going to fill in the blanks assuming a on-prem sql > lakehouse (sql endpoint) > semantic model, if this is your case the trick is to "Refresh SQL Endpoint" after you complete loading the data, there's two ways to do this programtically (see this article from our community colleague vojtechsima on this topic Programmatically Refresh & Sync SQL Analytics Endpoint Metadata in Microsoft Fabric ) or via new Refresh SQL Endpoint activity Refresh SQL Endpoint Activity - Microsoft Fabric | Microsoft Learn ... in!
So, I guess I would park any additional option here until you share some additional details on what's in between the semantic model and your on-prem sql database 😉 ... but the pattern I described is quite common, hence, this suggestion.
Hope this information helps, if so, a thumbs-up is always nice, looking forward for additional comments... all the best.
Hi alloowishus,
Thank you for reaching out to the Microsoft Fabric Community Forum. Also, thanks to svenchio, Vinodh247, for those inputs on this thread.Has your issue been resolved? If the response provided by the community member svenchio, Vinodh247, addressed your query, could you please confirm? It helps us ensure that the solutions provided are effective and beneficial for everyone.
Hope this helps clarify things and let me know what you find after giving these steps a try happy to help you investigate this further.
Thank you for using the Microsoft Community Forum.Hi alloowishus,
Just wanted to follow up. If the shared guidance worked for you, that’s wonderful hopefully it also helps others looking for similar answers. If there’s anything else you'd like to explore or clarify, don’t hesitate to reach out.Thank you.
