Forum Discussion

amaaiia's avatar
amaaiia
Skilled Sharer
2 years ago
Solved

Duplicated rows between notebook and SQL Endpoint

I'm having troubles when trying to count the number of rows in my table from Lakehouse.   If I count the rows from SQL Endpoint, I get 221555 rows, if I read the table from notebook and then I coun...
Data Engineering
  • Scott_Powell's avatar
    Scott_Powell
    2 years ago

    Hi all, there's apparently a bug where the metadata on which parquet file is the latest and greatest can get hosed up between the lakehouse SQL endpoint and the notebooks. I worked a ticket with Microsoft and they had me run the following code. After running it, wait 30 minutes or so to retry running the notebook. In my case this perfectly fixed the issue.

     

    mssparkutils.fs.unmount("/default", {"scope": "default_lh"})
    sc._jvm.com.microsoft.spark.notebook.common.trident.TridentRuntimeContext.reset()
    sc._jvm.com.microsoft.spark.notebook.common.trident.TridentRuntimeContext.personalizeSession()
     
    Hope this helps,
    Scott
amaaiia's avatar
amaaiia
Skilled Sharer
2 years ago

Hi,

I didn't find a solution. As I was in dev environment trying a demo I just deleted an laoded data again. I'm not having duplicates now. I hope this won't happen again.

  • MysticSapphire's avatar
    MysticSapphire
    Frequent Visitor
    2 years ago

    I'm experiencing a similar issue:
    My dataflow Gen2 is storing data in lakehouse. Once in a while the 'replace' table setting in the dataflow Gen2 doesn't seem to work and it results in having the same data copied twice. It only seems to be affecting smaller tables. If I delete the table it works for a few days but then suddenly there are duplicates again.