topic Re: Semantic model refreshes after Lakehouse update and data is not in SM in Pipelines

Semantic model refreshes after Lakehouse update and data is not in SM

amaaiia — Mon, 14 Oct 2024 07:58:28 GMT

I have a Data Pipeline that first removes some data from a Lakehouse and then adds new data in the Lakehouse through notebooks, and when this processing is finished, it finally executes the task Refresh Semantic Model to refresh the semantic model that reads the data from the Lakehouse.

An error is happening several times, after the refresh, I see that the semantic model has the data of the Lakehouse after removing data (begore adding new data). I've set a delay of 1 minute between the end of the notebook processing and the refresh of the model, to give the Lakehouse time to update itself. Even with the 1 minute delay, the error remains happening.

When I realize the error has happened (a few hours later), I manually refresh se SM and it reloads correctly what is in the lakehouse.

Why if the refresh task of the model is always executed after all the Pipeline flow is finished, the SM refreshes a previous state? Why does it take so long for the Lakehouse to be updated? The SM is refresehd after all Data Pipeline is executed, why is it retreiving a previous state?

Re: Semantic model refreshes after Lakehouse update and data is not in SM

jwinchell40 — Mon, 14 Oct 2024 13:05:37 GMT

amaaiia - I've noticed delays as well with data being available in the Lakehouse. Usually by the time my next Notebook starts up and the cluster warms up (using custom environment vs default pools) everything is there.

One thing we've turned off on some of our more complex Semantic Models running in Direct Lake mode was the feature that automatically checks for changes and refreshes the Lakehouse. The data we are pushing through our solution happens in batches we really only need to refresh the Semantic Model at the end of the process.

I would open a ticket with Fabric Support to see if anything else is impacting the latency (Capacity size, configuration, etc).

Jeremy

Re: Semantic model refreshes after Lakehouse update and data is not in SM

frithjof_v — Tue, 15 Oct 2024 22:11:23 GMT

It sounds like the reason is SQL Analytics Endpoint metadata sync delay:

SQL Analytics Endpoint performance : r/MicrosoftFabric (reddit.com)

Please create a support ticket to highlight the issue.

Re: Semantic model refreshes after Lakehouse update and data is not in SM

frithjof_v — Thu, 17 Oct 2024 04:52:20 GMT

Here is a fresh related discussion on Reddit:

https://www.reddit.com/r/MicrosoftFabric/s/aWKHWv3eWy