topic Re: Lakehouse - sync issues to SQL en`dpoint in Data Engineering

Lakehouse - sync issues to SQL en`dpoint

joakimfenno — Fri, 13 Sep 2024 09:45:18 GMT

I have a delta table in lakehouse which is populated by a notebook from another delta table

When I query the table ditrect I see records that I dont see when I query the table fromn the SQL endpoint

I have seen sync issues to the sql endpoint before (created objects in lakehouse appear in the sql endpoint after some time) but I have not seen this issue before. I was thinking that it could take a while before the sql endpoint is refreshed but it has now been a couple of hours.

Has anyone else seen this?

Re: Lakehouse - sync issues to SQL en`dpoint

digitalbrain — Fri, 13 Sep 2024 16:13:24 GMT

Hi @joakimfenno , thank you for raising this issue. I've seen the similar issue when accessing the data right after overwritting the Lakehouse delta table via Notebook. Sometimes, the SQL endpoint returns the old data regardless of the new data available in the delta table when it's read by the Lookup activity in the pipeline. This must be an issue at Fabric side that requires their attention to patch in the next releases.

Re: Lakehouse - sync issues to SQL en`dpoint

AndyDDC — Fri, 13 Sep 2024 17:13:26 GMT

Yes this is a known issue. Do you have a lot of partitions in your Delta tables? People have been reporting that the larger number of partitions they have, the longer the sync takes due to the service having to parse a large Delta log.

Re: Lakehouse - sync issues to SQL en`dpoint

frithjof_v — Sat, 14 Sep 2024 12:12:35 GMT

Here is some documentation regarding SQL Analytics Endpoint performance:

https://learn.microsoft.com/en-us/fabric/data-warehouse/sql-analytics-endpoint-performance

Also, if you by chance have used column name mapping to alter your table, it will stop syncing to the SQL Analytics Endpoint.

Re: Lakehouse - sync issues to SQL en`dpoint

digitalbrain — Sun, 15 Sep 2024 01:56:03 GMT

Thank you for providing the MS learn link.

"Under normal operating conditions, the lag between a lakehouse and SQL analytics endpoint is less than one minute."

The lag between lakehosue and SQL endpoint was showing the difference for me and MS Learn does confirm the lag as it takes around 1 minute to update the SQL endpoint. In my secnario, I needed to access the data right after updating the lakehouse table, hence I was getting the difference.

Re: Lakehouse - sync issues to SQL en`dpoint

Anonymous — Mon, 16 Sep 2024 07:52:46 GMT

Hi @joakimfenno ,

Thank you all for your attention to this case.

Here are a few potential reasons and solutions:

Under normal conditions, the lag between updates in the Lakehouse and the SQL endpoint should be less than a minute. If there are many partitions in your Delta tables, the sync might take longer due to the service parsing a large Delta log.

If you have used column name mapping to alter your table, it might stop syncing to the SQL endpoint.

You can try forcing a sync manually by updating the table on OneLake or waiting for the sync to complete automatically.

If you have any other questions please feel free to contact me.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

Re: Lakehouse - sync issues to SQL en`dpoint

Anonymous — Tue, 24 Sep 2024 07:01:42 GMT

Hi @joakimfenno ,

Is my follow-up just to ask if the problem has been solved?

If so, can you accept the correct answer as a solution or share your solution to help other members find it faster?

Thank you very much for your cooperation!

Best Regards,
Yang
Community Support Team

Re: Lakehouse - sync issues to SQL en`dpoint

joakimfenno — Tue, 24 Sep 2024 10:42:45 GMT

the sql endoint synced after two days, but I dont understand why/how

Re: Lakehouse - sync issues to SQL en`dpoint

Anonymous — Wed, 25 Sep 2024 08:05:28 GMT

Hi @joakimfenno ,

Sometimes, there is a delay in synchronization, albeit a long time.

If the issue has been resolved, can you please mark this post as "Answered"?

Best Regards,
Yang
Community Support Team

Re: Lakehouse - sync issues to SQL en`dpoint

joakimfenno — Wed, 25 Sep 2024 12:38:55 GMT

for some cases its hard to tell is cases are resolved or not 🙂

in my case the sql endpoint were in sync two days later - is that expected or a bug?

You wrote that "Under normal conditions, the lag between updates in the Lakehouse and the SQL endpoint should be less than a minute. If there are many partitions in your Delta tables, the sync might take longer due to the service parsing a large Delta log."

in my case we did not have any partitions

Re: Lakehouse - sync issues to SQL en`dpoint

smpa01 — Wed, 25 Sep 2024 20:23:22 GMT

@joakimfenno I am in the same boat as you. Can you please confirm me 1 thing if possible, after you perform the delta table operations, if you manually refresh following this, does the sql endpoint sync immediately ? src - https://learn.microsoft.com/en-us/fabric/data-warehouse/sql-analytics-endpoint-performance

Re: Lakehouse - sync issues to SQL en`dpoint

joakimfenno — Thu, 26 Sep 2024 05:13:43 GMT

Yes I tried that
I will try again next time this occurs and get back