topic Re: Microsoft Fabric: Data Duplicated Between SQL Endpoint and Notebook in Data Engineering

Microsoft Fabric: Data Duplicated Between SQL Endpoint and Notebook

davishoover00 — Thu, 23 May 2024 20:18:45 GMT

Hello all,

I currently bring data into Fabric through Dataflows using the ODBC connector. I land "delta" data in a separate staging lakehouse, and then I use a Notebook to update my production tables. This process is scheduled to occur every 4 hours. It brings in updated transactional data from our ERP, and updates the rows that have changed in the production table. When I query the SQL Endpoint I get accurate results with one entry per Internal ID of the most recent data. When I run the same query in a notebook I get duplicates of some (but not all) entries.

I ran into this issue before and thought it might pertain to the spark intelligent cache. I have set the cache to false, which didn't work. I also stumbled across this post about the same issue, and employed the solution in my update pipelines. Resetting the lakehouse mount points worked for a few weeks, but now the issue is back and I cannot figure out why this is occurring.

Query results of the SQL Endpoint:

Query results from Notebook:

Code to merge data:

Re: Microsoft Fabric: Data Duplicated Between SQL Endpoint and Notebook

Anonymous — Fri, 24 May 2024 10:51:47 GMT

Hi davishoover00
Thanks for using Fabric Community.
Apologize for the issue you are facing. The best course of action is to open a support ticket and have our support team take a closer look at it and get this resolved.

Please reach out to our support team so they can do a more thorough investigation on why this is happening: Link

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Hope this helps. Please let us know if you have any other queries.

Re: Microsoft Fabric: Data Duplicated Between SQL Endpoint and Notebook

Anonymous — Mon, 27 May 2024 16:13:11 GMT

Hi davishoover00
We haven’t heard from you on the last response and was just checking back to see if you got a chance to create a support ticket. If yes please share the details here. Otherwise, will respond back with the more details and we will try to help.
Thanks

Re: Microsoft Fabric: Data Duplicated Between SQL Endpoint and Notebook

frithjof_v — Sat, 01 Jun 2024 19:14:12 GMT

Sounds similar to the issue mentioned in this thread also:

https://community.fabric.microsoft.com/t5/Data-Engineering/Duplicated-Rows-In-Tables-Built-By-Notebook/m-p/3811276/highlight/true#M778

I hope this issue will get fixed soon.

Please create a support ticket if you have the time.

Re: Microsoft Fabric: Data Duplicated Between SQL Endpoint and Notebook

frithjof_v — Sun, 02 Jun 2024 06:50:46 GMT

Ref. also this post:

https://community.fabric.microsoft.com/t5/Data-Engineering/Fabric-Notebook-shows-wrong-of-rows-in-a-Lakehouse-table/m-p/3811408#M5880