Forum Discussion

alloowishus's avatar
alloowishus
Icon for Helper I rankHelper I
3 months ago
Solved

Regularily updating fact data refresh time

I have created some extensive pipeplines for copying data from our on prem SQL databases. I am doing some incremental loading, but also doing straight overwriting for some fairly large fact tables (a...
  • Vinodh247's avatar
    3 months ago

    What you are seeing is expected behaviour in fabric/pbi and it is not just “data copy complete- ready to query”.

    After your pipeline finishes, several backend operations still run before the semantic model becomes queryable: data commit to OneLake, delta/columnstore optimization, metadata synchronization, and most importantly semantic model refresh processing (including partition processing and index encoding). During this window, the model can temporarily reject connections, which is why users see “cannot connect”.

     

    There is no single flag in the pipeline that guarantees “PBI ready”. Instead, you need to track semantic model refresh completion, not pipeline completion.

     

    What to use:

    • Monitor semantic model refresh status in Fabric (or Power BI service) this is the source of truth

    • Use refresh history/REST API (Get Refresh Execution Details) to confirm status = Completed

    • If using pipelines, trigger and wait on a dataset refresh activity, not just data load

    • For large overwrite loads, consider incremental refresh/partitioning to avoid full reprocessing delays

    Your delay is coming from semantic model processing, not data movement. Treat dataset refresh completion as the real “done” signal, otherwise you will always see this lag.