Forum Discussion

amaaiia's avatar
amaaiia
Skilled Sharer
2 years ago
Solved

Duplicated rows between notebook and SQL Endpoint

I'm having troubles when trying to count the number of rows in my table from Lakehouse.   If I count the rows from SQL Endpoint, I get 221555 rows, if I read the table from notebook and then I coun...
data engineering
  • Scott_Powell's avatar
    Scott_Powell
    2 years ago

    Hi all, there's apparently a bug where the metadata on which parquet file is the latest and greatest can get hosed up between the lakehouse SQL endpoint and the notebooks. I worked a ticket with Microsoft and they had me run the following code. After running it, wait 30 minutes or so to retry running the notebook. In my case this perfectly fixed the issue.

     

    mssparkutils.fs.unmount("/default", {"scope": "default_lh"})
    sc._jvm.com.microsoft.spark.notebook.common.trident.TridentRuntimeContext.reset()
    sc._jvm.com.microsoft.spark.notebook.common.trident.TridentRuntimeContext.personalizeSession()
     
    Hope this helps,
    Scott
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi amaaiia 
We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks

amaaiia's avatar
amaaiia
Skilled Sharer
2 years ago

Hi,

I didn't find a solution. As I was in dev environment trying a demo I just deleted an laoded data again. I'm not having duplicates now. I hope this won't happen again.