Forum Discussion

amaaiia's avatar
amaaiia
Skilled Sharer
2 years ago
Solved

Duplicated rows between notebook and SQL Endpoint

I'm having troubles when trying to count the number of rows in my table from Lakehouse.   If I count the rows from SQL Endpoint, I get 221555 rows, if I read the table from notebook and then I coun...
Data Engineering
  • Scott_Powell's avatar
    Scott_Powell
    2 years ago

    Hi all, there's apparently a bug where the metadata on which parquet file is the latest and greatest can get hosed up between the lakehouse SQL endpoint and the notebooks. I worked a ticket with Microsoft and they had me run the following code. After running it, wait 30 minutes or so to retry running the notebook. In my case this perfectly fixed the issue.

     

    mssparkutils.fs.unmount("/default", {"scope": "default_lh"})
    sc._jvm.com.microsoft.spark.notebook.common.trident.TridentRuntimeContext.reset()
    sc._jvm.com.microsoft.spark.notebook.common.trident.TridentRuntimeContext.personalizeSession()
     
    Hope this helps,
    Scott
Scott_Powell's avatar
Scott_Powell
Advocate IV
2 years ago

Hi all, there's apparently a bug where the metadata on which parquet file is the latest and greatest can get hosed up between the lakehouse SQL endpoint and the notebooks. I worked a ticket with Microsoft and they had me run the following code. After running it, wait 30 minutes or so to retry running the notebook. In my case this perfectly fixed the issue.

 

mssparkutils.fs.unmount("/default", {"scope": "default_lh"})
sc._jvm.com.microsoft.spark.notebook.common.trident.TridentRuntimeContext.reset()
sc._jvm.com.microsoft.spark.notebook.common.trident.TridentRuntimeContext.personalizeSession()
 
Hope this helps,
Scott
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi Scott_Powell 
Thanks for sharing the solution here. Please continue using Fabric Community for any help regarding your queries.