Microsoft Fabric: Data Duplicated Between SQL Endpoint and Notebook

Hello all,

I currently bring data into Fabric through Dataflows using the ODBC connector. I land "delta" data in a separate staging lakehouse, and then I use a Notebook to update my production tables. This process is scheduled to occur every 4 hours. It brings in updated transactional data from our ERP,  and updates the rows that have changed in the production table. When I query the SQL Endpoint I get accurate results with one entry per Internal ID of the most recent data. When I run the same query in a notebook I get duplicates of some (but not all) entries.

I ran into this issue before and thought it might pertain to the spark intelligent cache. I have set the cache to false, which didn't work. I also stumbled across this post about the same issue, and employed the solution in my update pipelines. Resetting the lakehouse mount points worked for a few weeks, but now the issue is back and I cannot figure out why this is occurring.

Query results of the SQL Endpoint:

Query results from Notebook:

Code to merge data: