Microsoft Fabric: Data Duplicated Between SQL Endpoint and Notebook

[davishoover00]

Hello all,

I currently bring data into Fabric through Dataflows using the ODBC connector. I land "delta" data in a separate staging lakehouse, and then I use a Notebook to update my production tables. This process is scheduled to occur every 4 hours. It brings in updated transactional data from our ERP,  and updates the rows that have changed in the production table. When I query the SQL Endpoint I get accurate results with one entry per Internal ID of the most recent data. When I run the same query in a notebook I get duplicates of some (but not all) entries.

I ran into this issue before and thought it might pertain to the spark intelligent cache. I have set the cache to false, which didn't work. I also stumbled across this post about the same issue, and employed the solution in my update pipelines. Resetting the lakehouse mount points worked for a few weeks, but now the issue is back and I cannot figure out why this is occurring.

Query results of the SQL Endpoint:

Query results from Notebook:

Code to merge data:

[frithjof_v]

Sounds similar to the issue mentioned in this thread also:

https://community.fabric.microsoft.com/t5/Data-Engineering/Duplicated-Rows-In-Tables-Built-By-Notebo...

I hope this issue will get fixed soon.

Please create a support ticket if you have the time.

[frithjof_v]

Ref. also this post:

https://community.fabric.microsoft.com/t5/Data-Engineering/Fabric-Notebook-shows-wrong-of-rows-in-a-...

[Anonymous]

Hi @davishoover00 
Thanks for using Fabric Community.
Apologize for the issue you are facing. The best course of action is to open a support ticket and have our support team take a closer look at it and get this resolved.

Please reach out to our support team so they can do a more thorough investigation on why this is happening: Link 

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Hope this helps. Please let us know if you have any other queries.

[Anonymous]

Hi @davishoover00 
We haven’t heard from you on the last response and was just checking back to see if you got a chance to create a support ticket. If yes please share the details here. Otherwise, will respond back with the more details and we will try to help.
Thanks