Forum Discussion
Semantic Model Refresh Not Reflecting Latest Lakehouse Data using data pipeline
Hello All,
I'm encountering an unusual issue with one of the data pipelines I built, which extracts data from BigQuery into the Fabric Lakehouse. The pipeline uses a copy activity to move data from BigQuery to the Lakehouse, and this part works as expected—record counts match perfectly between both sources.
Once the BigQuery load completes, I trigger a semantic model refresh using the Semantic Model activity. The refresh completes successfully, but here's where the problem arises: the report still shows yesterday’s timestamp as the latest, even though the Lakehouse contains today’s data.
Interestingly, when I manually refresh the semantic model at 8 AM, it correctly picks up the latest changes. The scheduled refresh, however, runs daily at 2 AM—well after the Lakehouse has received the updated data. So it seems the Lakehouse has had sufficient time to sync metadata, yet the scheduled refresh doesn’t reflect the latest timestamp.
Has anyone experienced something similar or have insights into why the scheduled refresh might not be picking up the latest data?
Any help or suggestions are greatly appericited.
Here are the screen shots for
Here is the data pipeline:
Copy activity refresh: using service account ( from BQ to lakehosue)
Semantic model refresh: using serivice principle
18 Replies
Hi sumanthkakarla,
Are these semantic models using DirectLake, or are they using import mode/DirectQuery via the SQL endpoint?
If you're using the endpoint, try issuing a dummy SELECT 1 FROM table statement against the endpoint before refreshing the models, this should force the metadata sync.
I'd also suggest swotching to DIrectLake unless you have a business requirement not to, this bypasses the endpoint all together and personally I've found the sql endpoints to be immature and unreliable in some cases.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
Thank you tayloramy
We are using Import mode for these semantic models via the SQL endpoint, and there's a specific business reason for this setup. The models are being embedded into an external website using a service principal, and currently, Import mode is the only supported option for this authentication method.
To issue a dummy SELECT statement to trigger metadata refresh or schema binding, do I need to create a notebook and execute the dummy query from there? Or is there an alternative approach you'd recommend for this scenario?
Hi sumanthkakarla,
Gotcha, the embedding is a business reason that you cannot use directLake mode.
THe SQL endpoint in my experience needs some aggressive poking before it re-syncs. You can do this by using a script activity and selecting the top 1 row from the tables in your model, selecting from them will force the metadata sync.
You can also just refresh the models, wait 5 minutes, then refresh the models again. That is a messier solution, but should achieve the same goal.
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
Hi sumanthkakarla, I'm not sure that will work, try using the Script activity to issue a SQL statement against the SQL endpoint.
Using SparkSQL in a notebook doesn't actually interact with the SQL Endpoint, and it's that interaction that causes the endpoint to sync.If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
HI sumanthkakarla,
Darn, I was hoping forcing the sync would work.
Did you add a delay after the script before the model refresh?
sumanthkakarlaDarn.
I'm not able to replicate this issue in my environment, so I think that's all the help I can offer. Gave it a solid effort 😞
Maybe open a ticket with Microsoft and see what the support engineers say?
If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.
Hi sumanthkakarla ,
Thanks for reaching out to the Microsoft fabric community forum
Thanks for your prompt response
I wanted to follow up and confirm whether you’ve had the opportunity to review the information provided tayloramy . If you have any questions or need further clarification, please don’t hesitate to reach out.
Looking forward to your response.
Best regards,
Lakshmi.Hi v-lgarikapat
For us the first solution did not work, i need to work on the messier solution sugges by tayloramy
Will update you tomorrow if the other solution is working as expected.
